mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings

outleradam · 2009-12-31 20:33

mythicalLibrarian
http://wiki.xbmc.org/?title=MythicalLibrarian

Key features:

Episodes renamed to Show name.SxxExx (episode title).ext
Movies renamed to Movie (year).ext
Special provisions for naming sports and news which is compatible with XBMC.
Symlinks allow MythTV to maintain your library.
TvDb recognition of Show Name and Episode Name
Zap2it ID and fuzzy logic Title matching allows for perfect naming of Titles
Airdate and fuzzy logic episode matching allows for perfectly named episodes
User defined show name translations for improper guide data
User is notified at the end of each operation, XBMC is updated
Dynamic mount support with alternate move dir
mythicalSetup allows for easy installation and selection of options, while also making decisions based on your setup.
Can be run as MythTV job.
Scan mode will set the naming convention of your existing library
User jobs can be configured to run using the information mythicalLibrarian has collected

Code:

mythicalLibrarian myfile.ext "title to match" "subtitle to match"

--or--

mythicalLibrarian "path/to/mythtv_file.ext"

--or as a user job in mythtv--

mythicalLibrarian "%DIR%/%FILE%"

How to use:
mythicalLibrarian will generate usable content for XBMC out of your MythTV recorded libraries by utilizing the standard Showname.SxxExx (Episode Title).ext format.

run the mythicalSetup.sh program
run mythicalLibrarian --help

This will explain how to use everything.

It can be configured in the following ways

Move your files and create symlinks in place of the original so that MythTV can manage your library
Leave the original in place and create symlinks to your folder
Move files to a NAS and if the NAS is not present, move it to a local folder
After extinguishing all other resources, create a symlink to the original file in a local folder

I would like any and all problem reports for mythicalLibrarian to be filed here.

A problem report consists of a brief discription of the problem and a copy of a single debug log from output.log Please report anything you consider to be a problem as it will help out others and improve the development of the program. Before posting, please look at the log to see if it contains any valuable information.

The entire debug log is located in ~/.mythicalLibrarian/output.log and a single log starts with "NEW SEARCH INITIATED AT" and contains the words "DEBUG MODE ENABLED". Logs should be wrapped in [ code] [ /code] tags for clenliness.
Here is an example debug log:

Code:

@@@@@@@@@@@NEW SEARCH INITIATED AT Mon Sep 20 18:30:42 CDT 2010@@@@@@@@@@@@@@

Revision 600 Mon Sep 20 17:56:31 CDT 2010 on Linux

PERFORMING MAINTENANCE ROUTINE

REMOVING ORPHAN /home/mythtv/Videos/Episodes/SpongeBob SquarePants/Season 6/SpongeBob SquarePants S06E16 (Boating Buddies).txt

REMOVING ORPHAN /home/mythtv/Videos/Episodes/SpongeBob SquarePants/Season 6/SpongeBob SquarePants S06E10 (A Life in a Day).txt

REMOVING ORPHAN FOLDER:/home/mythtv/.mythicalLibrarian/mythicalSetup/Episodes/Mister Rogers Neighborhood/Season 7

REMOVING ORPHAN FOLDER:/home/mythtv/.mythicalLibrarian/mythicalSetup/Episodes/Mister Rogers Neighborhood

SEARCHING: www.TheTvDb.com SHOW NAME: Tosh.0 EPISODE: 

DATE: 2009-08-06 FILE NAME: /home/mythtv/Videos/1056_20100920180000.mpg

SEARCH FOUND:Tosh.0 ID#: 97731

MATCH!!!! Zap2itID:Tosh.0 ID.97731

###################DEBUG MODE ENABLED####################

MY USER NAME:mythtv-

LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.

INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/.mythicalLibrarian-

MOVE DIR:/home/mythtv/Videos/Episodes- USING SHOWNAME AS FOLDER:Enabled-

FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/mythtv/Videos/Episodes-

USE ORIGINAL DIR:Enabled NOTIFICATIONS:Enabled

INPUT SHOW NAME:Tosh.0- LOCAL SHOW NAME TRANSLATION:- ShowName:Tosh.0

DATABASE UPDATED:- TVDB LAST UPDATED:1285090977- CURRENT:1285026385-

RESOLVED SERIES ID:97731- RESOVED SHOW NAME:Tosh.0-

INPUT EPISODE NAME:- ABSOLOUTE EPISODE NUMBER:9- RESOLVED EPISODE NAME:Web Redemption Drunk Dunk-

SEASON:S01- EPISODE:E09- SYMLINK MODE:MOVE- FILESIZE: 0kB-

CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:4355428kB- WORKINGDIR:1 FREE:4355364kB-

MOVEDIRWRITABLE:1- FREE:4355428kB- ALTERNATEMOVEDIR:1- FREE:4355428kB-

PRIMARYSHOWDIRWRITABLE:1-  FREE:4355428kB-ALTERNATESHOWDIRWRITABLE:1- FREE:4355428kB-

PRIMARYMOVIEDIRWRITABLE:1- FREE:4355428kB- ALTERNATEMOVIEDIR:1- FREE:4355428kB-

DATABASE TYPE:schedulesdirect1-

 RECSTART:2010-09-20 18:00:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2009-08-06-

 PROGRAMID:EP011496300009- CHANNEL ID:1056- CATEGORY:Comedy- GOFORDOOVER:1-

 EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0

 ZAP2IT SERIES ID:1149630- MATCHED TVDB SERIES ID:97731-

PLOT: Daniel's Popcorn Challenge; Dizzy Dunk's web redemption; Tommy Chong.

####################END OF DEBUG LOG#####################

MOVING FILE: '/home/mythtv/Videos/1056_20100920180000.mpg' to '/home/mythtv/Videos/Episodes/Tosh.0/Season 1/Tosh.0 S01E09 (Web Redemption Drunk Dunk).mpg'

@@@@@@@@@@@@@OPERATION COMPLETE Mon Sep 20 18:30:47 CDT 2010 @@@@@@@@@@@@@@@@

Please report all errors you find.

I am subscribed to this thread and will be notified within about 15 minutes on my Android.
Last "stable" is here: http://mythicallibrarian.googlecode.com/...alSetup.sh
Newest version will be here: http://mythicallibrarian.googlecode.com/...alSetup.sh
I am accepting donations at my sourceforge page as a way to justify the time I spend on this project to my girlfriend: https://sourceforge.net/projects/mythicallibrari/

outleradam · 2010-01-03 20:58

Corrected problem: when a folder becomes disconnected the comskip tracking are all removed.

outleradam · 2010-01-05 07:34

Patch has been applied to allow users to use their local database even without new data from thetvdb. It's working now. replace mythicalLibrarian.sh and it will use your current database until TheTvDb comes back up online and mythicalLibrarian's database can be updated again.

This is a perfect time to explain the doover.sh as well. doover.sh is just what it is named, it's a do-over. It records the jobs that have been run and allows you to re-run them if they've failed.

Code:

sudo su mythtv

sudo chmod +x /home/mythtv/mythicalLibrarian/doover.sh

/home/mythtv/mythicalLibrarian/doover.sh

mythicalLibrarian will then delete the list, cycle through each and every failed job it has performed, and rebuild the list of failed jobs. This way if TheTvDb is down, it will be easy to redo the jobs. Also, if the information is not in TheTvDb, you can add it, then go back and re-run doover.sh.

Between the two of these, downtime of TheTvDb should mean minimal impact

**althekiller** · 2010-01-05 08:12

How many threads do you need for this thing?

outleradam · 2010-01-05 14:45

Two. Are you saying I should close down the MythSExx thread?

Bueller · 2010-01-12 12:02

Thanks for the great script, I've been using it for about 1 week. I've run across an issue where the search logic doesn't find the correct series. It's for the show "Castle".

MythTV runs the following after the show has been recorded:
mythicalLibrarian.sh "Castle" "A Rose for Everafter" "/Recordings/1151_20100111210000.mpg", if fails with the following output:

Code:

RECSTART:2010-01-11 21:00:00 MOVIEYEAR: SERIESDATE:2010-01-11

PROGRAMID:EP010855880022 ShowCategory:Crime STARS:0 RATING:0

PLOT: While investigating the death of a bridesmaid at a wedding, Castle runs into an old flame (Alyssa Milano).

COMMERCIAL DATA START:25020 STOP:38123

COMMERCIAL DATA START:67537 STOP:81274

COMMERCIAL DATA START:109253 STOP:125929

COMMERCIAL DATA START:149902 STOP:163637

COMMERCIAL DATA START:179625 STOP:188904

Castle

woot

FUZZY Exx NUMBER:

EPISODE:A Rose for Everafter ABSOLUTE NUMBER:

FILE SIZE:5659270kB

MOVEDIR FREE SPACE: 338262196kB- WRITABLE:1

ALTMOVEDIR FREE SPACE: 54666004kB- WRITABLE:1

MOVIEDIR FREE SPACE: 338262196kB- WRITABLE:1

ALTMOVIEDIR FREE SPACE: 54666004kB- WRITABLE:1

ERROR: INFORMATION COULD NOT BE OBTAINED

Manually looking up the series on thetvdb.com show the show name as "Castle (2009)".
Changing the line in doover.sh to: mythicalLibrarian.sh "Castle (2009)" "A Rose for Everafter" "/Recordings/1151_20100111210000.mpg" causes the script to find the show and complete successfully.

Any chance you can add some logic to the script on failure to look for series names with years appended to them.

outleradam · 2010-01-12 15:39

Doh! I left that woot in there...

I will need the single debug log entry from /mythicalLibrarian/output.log to do any troubleshooting. Also I will release an un"woot"ed version tonight.

outleradam · 2010-01-12 16:01

Actually, create a file in mythicalLibrarian/ called showtranslations. Then put the following line in it:

Castle = Castle (2009)

This is the built in option.

During the day at work I check this thread on my iPhone. I just saw a terminal output and asked for a debug log. A debug log is always the best.

As for looking up a non specified name, that would be bad. The fuzzy logic would eventually make a match of a show which is completely unrelated. If searching battlestar gallactica, it would search 4 different shows to make a match. If mythicalLibrarian were to pull the original airdate year and apply it to the show name it would only be valid for season 1.

Shows which do not conform to thetvdb can be taken up with your guide data provider. mythicalLibrarian will work with what it is supplied by showtranslations, the guide data, thetvdb and MythTV. The problem in this case is the guide data, however mythicalLibrarian supplies a workaround.

outleradam · 2010-01-12 19:58

I was just reviewing thetvdb.com and found that the XML contains a zap2it Id. I will look into using this tonight.

The basic flow I am envisioning would be alot of work and would severely slow down the execution of mythicalLibrarian on certain shows, but if possible it would be worth it.

Basically this check would involve getting info from mythtv, performing the seriesid search, then downlading and parsing the first result, verifying the zaptoit ID, if it does not pass, then downloading the next XML and parsing it.. This will result in creation of many unneeded databases. I will see what I can come up with

outleradam · 2010-01-13 06:13

I came up with a fix. Here's the proof:

Code:

/home/mythtv/mythicalLibrarian/mythicalLibrarian.sh "castle" "A Rose For Ever After" "/home/adam/Desktop/new file .ext"

Accessing MythTV DataBase:

RECSTART: MOVIEYEAR: SERIESDATE:

PROGRAMID: ShowCategory: STARS: RATING:0

PLOT: 

Searching TheTvDb FOR Zap2it SERIES ID MATCH SH1085588

Please add Castle ID 82607 Zap2it entries to theTvDb.com

Please add Lavender Castle ID 72403 Zap2it entries to theTvDb.com

Please add Queenie's Castle ID 74533 Zap2it entries to theTvDb.com

Please add Union Castle ID 75368 Zap2it entries to theTvDb.com

Please add Eureeka's Castle ID 78403 Zap2it entries to theTvDb.com

Please add The Toy Castle ID 74602 Zap2it entries to theTvDb.com

Please add Blanding's Castle ID 71008 Zap2it entries to theTvDb.com

Please add Takeshi's Castle ID 74787 Zap2it entries to theTvDb.com

FOUND MATCH: Castle (2009) Zap2it: SH1085588 TheTvDb: 83462

Please add Up the Elephant and Round the Castle ID 96601 Zap2it entries to theTvDb.com

Please add Castle Haven ID 70971 Zap2it entries to theTvDb.com

IDENTIFIED SERIES AS ID:83462 - Castle (2009)

FUZZY Exx NUMBER:22

EPISODE:A Rose For Ever After ABSOLUTE NUMBER:22 S02E12

FILE SIZE:0kB

MOVEDIR FREE SPACE: 139744kB- WRITABLE:1

ALTMOVEDIR FREE SPACE: 3060236kB- WRITABLE:1

MOVIEDIR FREE SPACE: 139744kB- WRITABLE:1

ALTMOVIEDIR FREE SPACE: 3060236kB- WRITABLE:1

SET TARGET DIR AS: /home/mythtv/NAS/Video/shows

FILE NAME EXISTS.  FILE WILL BE KNOWN AS: Castle 2009.S02E12 (A Rose For Ever After)-2

CREATING LINK IN LINK MODE

You can find the "unstable" version at sourceforge https://sourceforge.net/projects/mythicallibrari/files/

althekiller Team-XBMC Developer Posts: 4,702 Joined: May 2004 Reputation: 12	2010-01-05 08:12 Post: #4 How many threads do you need for this thing? Howto post your problem in a useful manner. #xbmc-linux on FreeNode XBMC online-manual, FAQ, search, forum rules, how to submit a bugreport.
	find quote