XBMC Community Forum
mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings - Printable Version

+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for XBMC (/forumdisplay.php?fid=116)
+--- Thread: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings (/showthread.php?tid=65644)



- GregoryK - 2010-03-21 03:44

I upgraded to SVN 285 and recorded another Cycling event (Milan-San Remo classics race). Here is the file name:

Code:
Cycling S2010E320 (Milan-San Remo Recorded 2010-03-20@12.00.00).mpg

Here is the NFO file:

Code:
<tvshow>
    <title> Recorded 2010-03-20@12.00.00 -Episode</title>
        <rating>0</rating>
    <plot>From Italy.</plot>
    <genre>Sports event</genre>
</tvshow>

(shouldn't it have the title "Milan-San Remo" in the <title> tag?)

/Greg


- outleradam - 2010-03-21 03:51

It should. I'm working on it now. It should also have a season and episode in the NFO.

Btw.. to remove disconnected folders from future searches run
Code:
mythicalLibrarian --doMaintenance
This is a new feature in SVN 296. getting closer to proper naming.


- GregoryK - 2010-03-21 04:20

Cool. What do you mean by "disconnected folders" in your last post?

Also, will there be a way to re-run the Sports "S0E0" results from prior versions?

/Greg


- outleradam - 2010-03-21 04:40

disconnected folders are those which were deleted by the user or contained on disconnected removable storage or on a NAS which is not connected. mythicalLibrarian will report those as "DISCONNECTED FOLDER: /path/to/file.ext" in the ~/.mythicalLibrarian/output.log file each time maintenance is performed on a disconnected folder. --doMaintenance is a maintenance mode which ignores safety features which are in place in case of folder disconnection.

all categoric naming files should be rerun with --doover.


Trying to get setup - okiekyle - 2010-03-21 04:57

Downloaded version: THU 4MAR2010 2127

I "think" I got everything setup right. When I run --mythicalDiagnostics the first 4 pass except for XBMC notifications which I expect since I'm using an XBox and it's disabled in the script anyway.

When I get to the mythicalLibrarian test I get the following:
Code:
Falsifying information. MythTV-Database is not used for testing.
Performing fuzzy logic lookup on non-MythTV-database file.
Please wait. It may take a moment to build database files
executing mythicalLibrarian testfile.ext 'mister rogers' ' show 1332'
testing for exit status 0
./mythicalLibrarian: line 1069: mythicalLibrarian: command not found
exited with status 127
OVERALL FAILURE

I also get a command not found on lines 964 and 969 when I try a --scan

any ideas?


- outleradam - 2010-03-21 19:11

could you tell me what is loacted in /usr/local/bin/mythicalLibrarian line 1069? It's not the same as mine. Are you running bash4+? what OS are you using? I probly need to add more dependency requirements. I need to know what's on those lines. I did not program in a exit 127 so something very weird is going on.


- okiekyle - 2010-03-21 20:15

I'm running MythBuntu 9.10
Not sure what version of bash I'm using. passwd says:
Code:
mythtv:x:103:104::/home/mythtv:/bin/bash

Line 1069 of the script says:
Code:
mythicalLibrarian testfile.ext "mister rogers" "show 1332"

I installed mythicalLibrarian /home/mythtv/mythicalLibrarian per the wiki - looks like that's not right anymore.

I moved it to /usr/local/bin and when I do --mythicalDiagnostics now I get:

Code:
mythtv@HMythTV-Backend:~$ mythicalLibrarian --mythicalDiagnostics
--You have entered diagnostic mode
mythicalLibrarian will now conduct a series of tests.
press any key to verify installed packages...
package 'mysql-server-5.0' is installed
package 'curl' is installed
package 'agrep' or 'tre-agrep' is installed
package 'libnotify-bin' is installed
Installed packages check ***PASSED***
Press any key to perform Desktop Message test...
Performing librarian-notify-send test
This test verifies that the message was sent to the DBUS.
This test does not verify the user has a valid desktop open.
Desktop Notifications test ***PASSED***
Press any key to perform XBMC Notification test...
Performing XBMC Notifications test
SENDING REQUESTED COMMANDS TO:192.168.2.41:8080
SENDING REQUESTED COMMANDS TO:HxBox01:8080
XBMC Communications ***FAILED***
Press any key to test file system permissions...
Testing file system permissions
USER SETTING     |FLAG|Folder name
MoveDir          |1   |/home/mythtv/Videos/Library/TVShows
AlternateMoveDir |1   |/home/mythtv/Episodes
PrimaryMovieDir  |1   |/home/mythtv/Videos/Library/Movies/NotRated
AlternateMovieDir|1   |/home/mythtv/Movies
PrimaryShowDir   |1   |/home/mythtv/Showings
AlternateShowDir |1   |/home/mythtv/Showings
mythicalLibrarian|1   |/home/mythtv/.mythicalLibrarian
Critical Filesystem checks ***PASSED***
press any key to test mythicalLibrarian...
Falsifying information. MythTV-Database is not used for testing.
Performing fuzzy logic lookup on non-MythTV-database file.
Please wait. It may take a moment to build database files
executing mythicalLibrarian testfile.ext 'mister rogers' 'show 1332'
testing for exit status 0
REMOVING ORPHAN FOLDER:./Episodes/
Accessing MythTV DataBase:
RECSTART: MOVIEYEAR: SERIESDATE:
PROGRAMID: ShowCategory: STARS: RATING:0
PLOT:
FUZZY LOGIC SHOW NAME:  ID:
stat: cannot stat `testfile.ext': No such file or directory
df: `./Episodes': No such file or directory
df: no file systems processed
df: `./Episodes': No such file or directory
df: no file systems processed
UNUSABLE SPACE-CHECK:./Episodes
UNUSABLE SPACE-CHECK:./Episodes
JOB: mythicalLibrarian 'testfile.ext' 'mister rogers' 'show 1332'
FAILSAFE FLAG WAS SET
ATTEPMTING SYMLINK TO FAILSAFE DIR
ln: creating symbolic link `/home/mythtv/FailSafe/.ext': File exists
FILE SYSTEM ERROR:FailSafeModeComplete
exited with status 1
OVERALL FAILURE
press any key to continue on to the final results...
  All tests complete            RESULTS
  Installed Packages:        ***PASSED***
  Notifications Test:        ***PASSED***
  XBMC Communications:       ***FAILED***
  File System Checks:        ***PASSED***
  mythicalTesting:           COULD NOT COMPLETE
  Overall mythicalReadiness: ***FAILURE***
-----End of diagnostics-----
mythtv@HMythTV-Backend:~$ ls -la
total 48
drwxr-xr-x 12 mythtv mythtv 4096 2010-03-21 13:04 .
drwxr-xr-x  4 root   root   4096 2010-03-16 10:28 ..
-rw-------  1 mythtv mythtv 2707 2010-03-21 09:37 .bash_history
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-16 11:09 .config
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 13:04 Episodes
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 12:59 FailSafe
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-16 11:09 .fontconfig
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 13:04 Movies
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 12:57 mythicalLibrarian
drwxr-xr-x  3 mythtv mythtv 4096 2010-03-21 13:04 .mythicalLibrarian
drwxr-xr-x  2 mythtv mythtv 4096 2009-10-28 15:30 .mythtv
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 13:04 Showings
-rw-r--r--  1 mythtv mythtv    0 2010-03-20 20:33 .sudo_as_admin_successful
drwxr-xr-x  1 root   root      0 2010-01-06 20:09 Videos

Interesting it say "removing orphan folder ./Episodes" then later complains about not finding the folder ./Episodes. As you can see when it's done, ./Episodes is still there so I don't know what's up with that.

Any ideas?


- Lepy - 2010-03-21 20:36

Just a quick question, outleradam.

I have 148 recorded episodes of Good Eats taking up a large amount of space on my mythbox.

I'd like to to use nuvexport to easily cut commercials and convert to xvid while outputting appropriate files names (Good.Eats.SXXEXX.avi). Then, transfer the episodes to my nas and get them all pretty and organized in my xbmc library.

I prefer the mythfrontend for watching most shows but for those that I would like to archive, I like xbmc. Will mythicalLibrarian or MythSExx be the better option? Will either option allow nuvexport to export a cut and converted episode with its proper season and episode tag?


- outleradam - 2010-03-21 21:08

^^I would use mythicalLibrarian. mythSExx pretty much just does episodes. mythSExx is a very very scaled down version with less functionality and is intended to be run "mythSExx 'title' 'subtitle' 'path/to/file.ext'" while mythicalLibrarian is intended to be a full cataloguing solution. mythSExx would be a good option for those not running mythTV who have that information available.

okiekyle Wrote:I'm running MythBuntu 9.10
Not sure what version of bash I'm using. passwd says:
Code:
mythtv:x:103:104::/home/mythtv:/bin/bash
Line 1069 of the script says:
Code:
mythicalLibrarian testfile.ext "mister rogers" "show 1332"
I installed mythicalLibrarian /home/mythtv/mythicalLibrarian per the wiki - looks like that's not right anymore.

I moved it to /usr/local/bin and when I do --mythicalDiagnostics now I get:

Code:
mythtv@HMythTV-Backend:~$ mythicalLibrarian --mythicalDiagnostics
--You have entered diagnostic mode
mythicalLibrarian will now conduct a series of tests.
press any key to verify installed packages...
package 'mysql-server-5.0' is installed
package 'curl' is installed
package 'agrep' or 'tre-agrep' is installed
package 'libnotify-bin' is installed
Installed packages check ***PASSED***
Press any key to perform Desktop Message test...
Performing librarian-notify-send test
This test verifies that the message was sent to the DBUS.
This test does not verify the user has a valid desktop open.
Desktop Notifications test ***PASSED***
Press any key to perform XBMC Notification test...
Performing XBMC Notifications test
SENDING REQUESTED COMMANDS TO:192.168.2.41:8080
SENDING REQUESTED COMMANDS TO:HxBox01:8080
XBMC Communications ***FAILED***
Press any key to test file system permissions...
Testing file system permissions
USER SETTING     |FLAG|Folder name
MoveDir          |1   |/home/mythtv/Videos/Library/TVShows
AlternateMoveDir |1   |/home/mythtv/Episodes
PrimaryMovieDir  |1   |/home/mythtv/Videos/Library/Movies/NotRated
AlternateMovieDir|1   |/home/mythtv/Movies
PrimaryShowDir   |1   |/home/mythtv/Showings
AlternateShowDir |1   |/home/mythtv/Showings
mythicalLibrarian|1   |/home/mythtv/.mythicalLibrarian
Critical Filesystem checks ***PASSED***
press any key to test mythicalLibrarian...
Falsifying information. MythTV-Database is not used for testing.
Performing fuzzy logic lookup on non-MythTV-database file.
Please wait. It may take a moment to build database files
executing mythicalLibrarian testfile.ext 'mister rogers' 'show 1332'
testing for exit status 0
REMOVING ORPHAN FOLDER:./Episodes/
Accessing MythTV DataBase:
RECSTART: MOVIEYEAR: SERIESDATE:
PROGRAMID: ShowCategory: STARS: RATING:0
PLOT:
FUZZY LOGIC SHOW NAME:  ID:
stat: cannot stat `testfile.ext': No such file or directory
df: `./Episodes': No such file or directory
df: no file systems processed
df: `./Episodes': No such file or directory
df: no file systems processed
UNUSABLE SPACE-CHECK:./Episodes
UNUSABLE SPACE-CHECK:./Episodes
JOB: mythicalLibrarian 'testfile.ext' 'mister rogers' 'show 1332'
FAILSAFE FLAG WAS SET
ATTEPMTING SYMLINK TO FAILSAFE DIR
ln: creating symbolic link `/home/mythtv/FailSafe/.ext': File exists
FILE SYSTEM ERROR:FailSafeModeComplete
exited with status 1
OVERALL FAILURE
press any key to continue on to the final results...
  All tests complete            RESULTS
  Installed Packages:        ***PASSED***
  Notifications Test:        ***PASSED***
  XBMC Communications:       ***FAILED***
  File System Checks:        ***PASSED***
  mythicalTesting:           COULD NOT COMPLETE
  Overall mythicalReadiness: ***FAILURE***
-----End of diagnostics-----
mythtv@HMythTV-Backend:~$ ls -la
total 48
drwxr-xr-x 12 mythtv mythtv 4096 2010-03-21 13:04 .
drwxr-xr-x  4 root   root   4096 2010-03-16 10:28 ..
-rw-------  1 mythtv mythtv 2707 2010-03-21 09:37 .bash_history
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-16 11:09 .config
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 13:04 Episodes
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 12:59 FailSafe
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-16 11:09 .fontconfig
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 13:04 Movies
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 12:57 mythicalLibrarian
drwxr-xr-x  3 mythtv mythtv 4096 2010-03-21 13:04 .mythicalLibrarian
drwxr-xr-x  2 mythtv mythtv 4096 2009-10-28 15:30 .mythtv
drwxr-xr-x  2 mythtv mythtv 4096 2010-03-21 13:04 Showings
-rw-r--r--  1 mythtv mythtv    0 2010-03-20 20:33 .sudo_as_admin_successful
drwxr-xr-x  1 root   root      0 2010-01-06 20:09 Videos
Interesting it say "removing orphan folder ./Episodes" then later complains about not finding the folder ./Episodes. As you can see when it's done, ./Episodes is still there so I don't know what's up with that.

Any ideas?

It looks like you're getting some sort of permission errors. try the following:
Code:
rm -Rf ~/.mythicalLibrarian #to remove mythicalLibrarian's database and setup
mythicalLibrarian --update #to create a new working folder
cd ~/.mythicalLibrarian/mythicalSetup #to start the setup process
sudo mythicalLibrarian --update #to resetup mythicalLibrarian
you should see this:
Code:
mythicalLibrarian will now conduct mythicalDiagnostics
Press any key to continue to online testing....

--You have entered diagnostic mode
mythicalLibrarian will now conduct a series of tests.
press any key to verify installed packages...

package 'mysql-server-5.0' is installed
package 'curl' is installed
package 'agrep' or 'tre-agrep' is installed
package 'libnotify-bin' is installed
Installed packages check ***PASSED***
Press any key to perform Desktop Message test...

Performing librarian-notify-send test
This test verifies that the message was sent to the DBUS.
This test does not verify the user has a valid desktop open.
Desktop Notifications test ***PASSED***
Press any key to perform XBMC Notification test...

Performing XBMC Notifications test
SENDING REQUESTED COMMANDS TO:192.168.1.110:8080
XBMC Communications ***PASSED***
Press any key to test file system permissions...

Testing file system permissions
USER SETTING     |FLAG|Folder name
MoveDir          |1   |/home/mythtv/Episodes
AlternateMoveDir |1   |/home/mythtv/Episodes
PrimaryMovieDir  |1   |/home/mythtv/Movies
AlternateMovieDir|1   |/home/mythtv/Movies
PrimaryShowDir   |1   |/home/mythtv/Showings
AlternateShowDir |1   |/home/mythtv/Showings
mythicalLibrarian|1   |/home/adam/.mythicalLibrarian
Critical Filesystem checks ***PASSED***
press any key to test mythicalLibrarian...

Falsifying information. MythTV-Database is not used for testing.
Performing fuzzy logic lookup on non-MythTV-database file.
Please wait. It may take a moment to build database files
executing mythicalLibrarian testfile.ext 'mister rogers' 'show 1332'
testing for exit status 0
Accessing MythTV DataBase:
RECSTART: MOVIEYEAR: SERIESDATE:
PROGRAMID: ShowCategory: STARS: RATING:0
PLOT:
FUZZY LOGIC SHOW NAME:  ID:
JOB: mythicalLibrarian 'testfile.ext' 'mister rogers' 'show 1332'
TARGET SET:./Episodes/
exited with status 0
press any key to continue on to the final results...

  All tests complete            RESULTS
  Installed Packages:        ***PASSED***
  Notifications Test:        ***PASSED***
  XBMC Communications:       ***PASSED***
  File System Checks:        ***PASSED***
  mythicalTesting:           ***PASSED***
  Overall mythicalReadiness: ***PASSED***
-----End of diagnostics-----
Installation and tests completed successfully
MythTV job not added because mythicalLibrarian already exists in slot 1
permissions were set for user: adam
RSS Feed will be located at http://192.168.1.133/mythical-rss/rss.xml
mythicalLibrarian is located in /usr/local/bin
'mythicalLibrarian --help' for more information
Done.

Thetvdb.com is down right now..http://downforeveryoneorjustme.com/www.thetvdb.com

so unless mythicalLibrarian has a local database to work with, it will assign a s0e0 to every episode. That's the great part though.. I didn't know the tvdb was down. mythicalLibrarian maintains a local database which it works with. When mythicalLibrarian encounters an error like that, it will put the file into the doover que. Files can be rerun with sudo mythicalLibrarian --doover at a later time for reprocessing.


- am4c130d - 2010-03-21 21:40

Last update was March 20th - but a long time between updates. All databases cleaned out so no issues caused by stale data

Couple of bugs.

Line 733 - missing "echo" before -e in adding season info to NFO - trivial.

When I try to move shows that have episode info (i.e. EPXXXX) but are not on tvdb.org, instead of moving the file to tvrootdir/showname/showname_S0E0 (data).mpg, mythicallibrarian omits the the showname directory. The tvshow.nfo and showname.nfo are created and seem fine, the problem is they're in the "root" tv directory, so tvshow.nfo is not show specific, and of it were, would be overwritten with each new show. I would personally prefer it to create a showname directory, and drop all the shows in to that directory - thus tvshow.nfo would be correct, and it would create less clutter.

That aside, mythicallibrarian is getting better and better - appreciate the hard work you are doing for those of us less able...

Thanks


- GregoryK - 2010-03-21 21:49

Quote: Also, will there be a way to re-run the Sports "S0E0" results from prior versions?
outleradam Wrote:all categoric naming files should be rerun with --doover.

The missing Cycling shows are listed in my doover.sh.old file but not my doover.sh file. Should I rename doover.old to doover before running --doover when you've finalized the categorical functionality?

/Greg


- outleradam - 2010-03-21 22:21

^^ I would like you to update to SVN 299 and test please. I was working on it last night with news programming and it seems to work exactly as intended.

am4c130d Wrote:Last update was March 20th - but a long time between updates. All databases cleaned out so no issues caused by stale data

Couple of bugs.

Line 733 - missing "echo" before -e in adding season info to NFO - trivial.

When I try to move shows that have episode info (i.e. EPXXXX) but are not on tvdb.org, instead of moving the file to tvrootdir/showname/showname_S0E0 (data).mpg, mythicallibrarian omits the the showname directory. The tvshow.nfo and showname.nfo are created and seem fine, the problem is they're in the "root" tv directory, so tvshow.nfo is not show specific, and of it were, would be overwritten with each new show. I would personally prefer it to create a showname directory, and drop all the shows in to that directory - thus tvshow.nfo would be correct, and it would create less clutter.

That aside, mythicallibrarian is getting better and better - appreciate the hard work you are doing for those of us less able...

Thanks
Thanks. I don't hear many complements about the work I've put into the program.

I committed about 10 changes last night relating to shows and categoic naming. Please run --undo (as the user or users who initiated the initial scan), --update, then --scan as mythtv. If those issues are not resolved, then I would like to know about it. SVN 299 is current. I will be searching for more bugs today.

So far the only thing I'm seeing is that News requires recorded on date to be used instead of original air date because all shows will be categorized as the 2006 premiere of Headline News.


So far under categoric naming: News, Sports event, Sports magazine.
Any other suggestions?


- Lepy - 2010-03-22 01:18

I am really digging the functionality and ease of use of this program. Thanks for putting it together!

Its been very accurate on the files I've tried...when thetvbd isn't down!

Back to my original question though...now that I have a folder filled with properly tagged and cutlisted mpgs [e.g. "Good Eats S05E06 (Berry From Another Planet).mpg"], what would be the best way to cut commercials and convert to xvid keeping the name?

Is there anyway I can now use the new name for the out put of a nuvexport file? In .nuvexportrc, I've tried a few combinations of filename=, but there doesn't seeme to be a variable for original filename.

I could manually run ffmpeg...but the thought of doing it 148 times doesn't sound like much fun.


- outleradam - 2010-03-22 02:28

you can set up your ffmpeg command as a user job in mythicalLibrarian.

edit /usr/local/bin/mythicalLibrarian and scroll down until you find user jobs. then put in the command after #Insert Custom User Job here

like this:
Code:
RunJob () {
     case $jobtype in
#Successful Completion of mythicalLibrarian
         LinkModeSuccessful|MoveModeSuccessful)
             echo "SUCCESSFUL COMPLETEION TYPE: $jobtype"
             #Insert Custom User Job here
             ffmpeg $InputPath
             #
             exit 0
             ;;
When you run in --scan mode, the last thing mythicalLibrarian does before scanning another file will be to call "ffmpeg /path/to/file.ext" . I'm not sure how to call ffmpeg, I've never run it before. $InputPath is the file being worked on.

Is this what you were looking for?


- GregoryK - 2010-03-22 02:35

outleradam:

OK, running svn 299, getting some weird output for --doover. It seems to be stuck in a loop re-doing the files in the doover file, for one thing. For another, there are permissions errors on several of the Sports showings (this may well be a problem on my end, but I don't understand why those particular files would have permissions issues and not the other files.

Pastebin of the terminal output (so far, it still seems to be running): http://pastebin.com/wzn20uGw

Pastebin of my doover.sh file: http://pastebin.com/1mncTxaB

Thanks as always for the support!

/Greg