Kodi Community Forum
mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings (/showthread.php?tid=65644)



- outleradam - 2010-08-16

Sorry it took me so long to respond. I saw your post yesterday on my phone, but I didn't get around a computer to type anything.

It looks like the /Showings file was correctly classified as a showing. On newer versions, after revision ~400, mythicalLibrarian will name a showing "recorded" if it is a true showing without information or will put a blurb in the title "Episode in --doover que" if it is a show with information.

Shows with information "epi-shows" are created when one of the following occur: A. Lack of connectivity to TheTvDb and no database entries on the show. B. information is not available on TheTvDb. C. Information was added within the last 18 hours since mythicalLibrarian last synchronized it's database to TheTVDb.

If the show DOES contain information, you can utilize the --doover function to rename the files

Code:
sudo mythicalLibrarian --doover

I've been considering adding a function to automatically run the doover que at 0300AM every morning during the average downtime. The problem with this is that some people may not allow XBMC to sleep at night so they may wake up to notifications of new shows being added to their library every night. So, for now --doover is manual.

As for the breakdown of the naming convention, it should be something like this:
/Episodes/Big Brother/Season 12/Big Brother S12E09 (Episode 9).mpg
that's:
/Episodes/Title/Season/Title SeasonXXEpisodeXX (TheTvDb subtitle).file extension

The naming convention is set so as to encorperate the optimal amount of information to make it human readable as well as machine readable/easily searchable. I believe the /season/ folder was requested a few months ago to make it DVD ready as well.

For Showings, the files are named with the S0E0 naming convention. The S0E0 is the "key" to insert the show into the XBMC library when it is unknown. There is no Season 0 Episode 0 on any known show so it is a universal way to add something to the library. Also, XBMC understands that it is not to look up that show to verify in the TVDB because it is accompanied with a NFO file which specifies all known information about the showing.


As for general maintenance of mythicalLibrarian since it has been a while since you upgraded, the database may have changed. It would be a good idea to flush the mythicalLibrarian database and take advantage of the newer naming conventions by doing the following (assuming MythTV still has information on all of the shows in your library).

Code:
sudo mythicalLibrarian --update
sudo mythicalLibrarian --doMaintenance
sudo mythicalLibrarian --undo
mythicalLibrarian --scan mpg /path/to/your/video_files/
This will pretty much reset everything. Then go to XBMC>Settings>Video>clean video library to remove any recordings which are no longer valid.



The output.log file contains all of the input for decisions made by mythicalLibrarian. Without ~/output.log entries I cannot tell you much more.


- outleradam - 2010-08-19

Any problems? Just post.


- outleradam - 2010-08-22

Let me know. SVN will become stable tonight, assuming the fresh installation I just performed on my media center goes smoothly. I have a few extra setup features in mind relating to mythicalSetup permissions and I want to make sure everything is working well before continuing on.

Please let me know about any ol' random error if you receive any and post a log on the file.


- GregoryK - 2010-08-24

Adam:

My only problem/issue right now is that I can't get mythicalLibrarian to recognize the Top Gear episodes that I record from BBC America, so they all get tagged as "Showings" not "Episodes.

The show is recorded as "Top Gear" by Myth, and includes program details. The show is listed in thetvdb under the same name: <http://thetvdb.com/index.php?tab=series&id=74608&lid=7>

Here's the output of 'ls /home/users/mythtv/.mythicalLibrarian/Top*':
Code:
ls Top*
Top Gear:
current.time  lastupdated.time  Top Gear.actualEname.txt  Top Gear.Ename.txt  Top Gear.E.txt  Top Gear.FAired.txt  Top Gear.S.txt  Top Gear.xml  Top Gear.zap2it.txt

Top Gear Australia:
current.time  lastupdated.time  Top Gear Australia.actualEname.txt  Top Gear Australia.Ename.txt  Top Gear Australia.E.txt  Top Gear Australia.FAired.txt  Top Gear Australia.S.txt  Top Gear Australia.xml  Top Gear Australia.zap2it.txt

Top Gear classic:
current.time  lastupdated.time  Top Gear classic.actualEname.txt  Top Gear classic.Ename.txt  Top Gear classic.E.txt  Top Gear classic.FAired.txt  Top Gear classic.S.txt  Top Gear classic.xml  Top Gear classic.zap2it.txt

Top Gear Russia:
current.time  lastupdated.time  Top Gear Russia.actualEname.txt  Top Gear Russia.Ename.txt  Top Gear Russia.E.txt  Top Gear Russia.FAired.txt  Top Gear Russia.S.txt  Top Gear Russia.xml  Top Gear Russia.zap2it.txt

Top Gear Xtra:
current.time  lastupdated.time  Top Gear Xtra.actualEname.txt  Top Gear Xtra.Ename.txt  Top Gear Xtra.E.txt  Top Gear Xtra.FAired.txt  Top Gear Xtra.S.txt  Top Gear Xtra.xml  Top Gear Xtra.zap2it.txt

Any suggestions on how to get mythicalLibrarian to recognize the show properly? Everything else seems to work fine that I have thrown at it.

Thanks!

/Greg


- outleradam - 2010-08-25

Please post the output of ~/.mythicalLibrarian/output.log. Just post one entry please. Not the entire log. I just need a debug log.


- tom_a_sparks - 2010-08-25

I have just downloaded a copy of mythicalLibrarian svn rev 508
mythicalLibrarian is not using my setting Sad
see config from myth*.sh and see the debug log
Code:
#SYMLINK has 3 modes.  MOVE|LINK|Disabled: Default=MOVE
#Create symlink in original dir from file after 'MOVE' | Do not move, just create a sym'LINK' | move the file, symlinking is 'Disabled'
SYMLINK='LINK'
#MoveDir is the folder which mythicalLibrarian will move the file.  No trailing / is accepted eg. "~/videos"
MoveDir="~/Videos/TV_Shows"  #<------THIS VALUE MUST BE SET-------
#AlternateMoveDir will act as a seccondary MoveDir if the primary MoveDir fails.  No trailing / is accepted eg. "~/videos"
AlternateMoveDir=
#If UseOriginalDir is Enabled, original dir will override MoveDir.  Useful for multiple recording dirs.
#UseOriginalDir will separate generic shows from episodes and movies. Enabled|Disabled
UseOriginalDir=Disabled
#When Enabled, mythicalLibrarian will move the file to a folder of the same name as the show. This is not affected by UseOriginalDir. Enabled|Disabled
UseShowNameAsDir=Enabled
#Internet access Timeout in seconds: Default Timeout=50 (seconds)
Timeout=50
#Update database time in secconds, Longer duration means faster processing time and less strain on TheTvDb. Default='84000' (1 day)
UpdateDatabase=1
#mythicalLibrarian working file dir: Default=~/mythicalLibrarian (home/username/mythicalLibraian)
mythicalLibrarian=~/.mythicalLibrarian
#FailSafe mode will enable symlinks to be formed in FailSafeDir if the move or symlink operation fails. Enabled|Disabled
FailSafeMode=Disabled
#FailSafeDir is used when the file cannot be moved to the MoveDir. FailSafe will not create folders. eg. /home/username
FailSafeDir="/home/mythtv/FailSafe"  #<------THIS VALUE MUST BE SET-------
#DirTracking will check for and remove the folders created by mythicalLibrarian
DirTracking=Enabled
#the following line contains the API key from www.TheTvDb.Com. Default project code: 6DF511BB2A64E0E9
APIkey=0F3A338F0FCB29A2
#Language setting
Language=en
  
###Database settings###
#MythTV MYSQL access allows addition of movies, comskip data, and improves accuracy of episode recognition.
#Database access Enabled|Disabled
Database=Enabled
#Database Type Default=MythTV
DatabaseType=MythTV
#Guide data type: SchedulesDirect currently covers DataDirect, MC2XML and others.  NoLookup will bypass tvdb scanning
#Options: SchedulesDirect|NoLookup
GuideDataType=SchedulesDirect

#Primary Movie Dir. mythicalLibrarian will attempt to move to this dir first. No trailing / is accepted eg. "~/videos"
PrimaryMovieDir="~/Videos/Movies" #<------THIS VALUE MUST BE SET-------
#AlternateMoveDir will act as a Seccondary move dir if the primary moive dir fails
AlternateMovieDir=""
#ShowStopper = Enabled prevents generic shows and unrecognized episodes from being processed
ShowStopper="Enabled"
#Primary Show Dir. mythicalLibrarian will attempt to move to this dir first. No trailing / is accepted eg. "~/videos"
PrimaryShowDir="~/Videos/Other_Shows" #<------THIS VALUE MUST BE SET-------
#AlternateMoveDir will act as a Seccondary move dir if the primary moive dir fails
AlternateShowDir=""
#CommercialMarkup will generate comskip files for recordings when they are moved. Enabled|Disabled
CommercialMarkup=Enabled
#CommercialMarkupCleanup will execute a maintenance routine which will remove comskip files if they are not needed
CommercialMarkupCleanup=Enabled

Code:
###################DEBUG MODE ENABLED####################
MY USER NAME:mythtv-box-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv-box/.mythicalLibrarian-
MOVE DIR:./Episodes- USING SHOWNAME AS FOLDER:Enabled-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:./Episodes-
USE ORIGINAL DIR:Enabled NOTIFICATIONS:Disabled DEBUG MODE:Enabled-
INPUT SHOW NAME:mister rogers- LOCAL SHOW NAME TRANSLATION:- ShowName:mister rogers
DATABASE UPDATED:- TVDB LAST UPDATED:1282803534- CURRENT:1282735074-
RESOLVED SERIES ID:77750- RESOVED SHOW NAME:Mister Rogers' Neighborhood-
INPUT EPISODE NAME:show 1332- ABSOLOUTE EPISODE NUMBER:462- RESOLVED EPISODE NAME:Show 1332-
SEASON:S07- EPISODE:E07- SYMLINK MODE:MOVE- FILESIZE: 0kB-
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:6337380kB- WORKINGDIR:1 FREE:6337376kB-
MOVEDIRWRITABLE:1- FREE:6337508kB- ALTERNATEMOVEDIR:1- FREE:6337488kB-
PRIMARYSHOWDIRWRITABLE:1-  FREE:6337380kB-ALTERNATESHOWDIRWRITABLE:1- FREE:6337380kB-
PRIMARYMOVIEDIRWRITABLE:1- FREE:6337388kB- ALTERNATEMOVIEDIR:1- FREE:6337388kB-
DATABASE TYPE:tv_grab_au-
RECSTART:- MOVIE YEAR:- ORIGINAL SERIES DATE:-
PROGRAMID:- CHANNEL ID:- CATEGORY:- GOFORDOOVER:1-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:- STARS: RATING:0
ZAP2IT SERIES ID:- MATCHED TVDB SERIES ID:-
PLOT:
####################END OF DEBUG LOG#####################
CREATING FOLDER: ./Episodes/Mister Rogers Neighborhood



- outleradam - 2010-08-25

I'm not really sure of what the problem is which you are describing. I need a bit more detail of what it is not doing. That debug log shows me that mythicalDiagnostics ran fine. It does not show me the problem which you are describing.

mythicalLibrarian is designed to take into account dynamic disks, flash drives, and disconnected network attatched storage. If the folder which the file resides does not exist, mythicalLibrarian will not remove symlinks. The folder must exist and the file must not exist for mythicalLibrarian to make the decision to remove the links and additional files.

Set up your alternate move, movie and show dir.
Disable showstopper to enable generic show recognition... unrecognized episodes are processed into the --doover que for later processing.

I hope some of what I said helps. I need more information to figure out what the problem is though.


Re: Top Gear - GregoryK - 2010-08-25

outleradam Wrote:Please post the output of ~/.mythicalLibrarian/output.log. Just post one entry please. Not the entire log. I just need a debug log.

Here you go, from the last recording:
Code:
@@@@@@@@@@@@@OPERATION COMPLETE Mon Aug 23 10:22:01 PDT 2010 @@@@@@@@@@@@@@@@
@@@@@@@@@@@NEW SEARCH INITIATED AT Mon Aug 23 19:23:09 PDT 2010@@@@@@@@@@@@@@
Revision 506 Mon Aug 16 22:00:36 PDT 2010
PERFORMING MAINTENANCE ROUTINE
SEARCHING: www.TheTvDb.com SHOW NAME: Top Gear EPISODE:
FILE NAME: /var/lib/mythtv/recordings/1162_20100823180000.mpg
SEARCH FOUND:Top Gear ID#: 74608
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear
MATCH!!!! Zap2itID:Top Gear ID.74608
SEARCH FOUND:Top Gear classic ID#: 142961
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear classic
SEARCH FOUND:Top Gear Russia ID#: 87701
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear Russia
SEARCH FOUND:Top Gear Australia ID#: 83234
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear Australia
SEARCH FOUND:Top Gear Xtra ID#: 83407
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear Xtra
DEFINED ABSOLOUTE EPISODE NUMBER:
WARNING:SHOW WAS CONSIDERED AN EPISODE BUT INFORMAITON WAS NOT OBTAINABLE
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
HANDLING EPISODE AS GENERAL SHOWING AND DUMPING INTO --doover USER JOB
GENERIC GUIDE DATA WAS SUPPLIED TYPE: Series With Episode Data- Top Gear,
WARNING:%%%%%%%%%%PROGRAM GUIDE DATA IS NOT COMPLETE%%%%%%%%%%%%%%%%%%%%
MOVING FILE: '/var/lib/mythtv/recordings/1162_20100823180000.mpg' to '/var/lib/mythtv/recordings/Showings/Top Gear/Top Gear S0E0 ( Recorded [email protected] on 1162).mpg'
@@@@@@@@@@@@@OPERATION COMPLETE Mon Aug 23 19:23:20 PDT 2010 @@@@@@@@@@@@@@@@

Thanks!

/Greg


- outleradam - 2010-08-25

I knew that the minute I disabled debug logging after the log exceeds 50K, it would be needed again. I will do an update tonight to try to resolve the logging issue. Debug logging will now be the default again. I'll post again here once I've enabled debug logging. Maybe it should be for 50,000K worth of logging after the last update.


- tom_a_sparks - 2010-08-26

when I run mythicalLibrary --scan nuv /media/disk
the episodes, Movies, Showings are created in /media/disk
when I run mythicalLibrary --mythicalDiagnostics
the episodes, movies, showing are created in ~/.mythicalLibrarian/mythicalSetup
Code:
Testing file system permissions
USER SETTING     |FLAG|Folder name
MoveDir          |1   |./Episodes
AlternateMoveDir |1   |./Episodes
PrimaryMovieDir  |1   |./Movies
AlternateMovieDir|1   |./Movies
PrimaryShowDir   |1   |./Showings
AlternateShowDir |1   |./Showings
mythicalLibrarian|1   |/home/mythtv-box/.mythicalLibrarian
Critical Filesystem checks ***PASSED***
I have changed all of them eg: ./Episodes to ~/Videos/TV_Shows, but they are not show up


- outleradam - 2010-08-26

That is normal. You are using the use original dir option. The dir is created by the location of which mythicalLibrarian is called. In diagnostic mode it calls a fake file in the current dir.


- outleradam - 2010-08-26

GregoryK Wrote:Here you go, from the last recording:
Code:
@@@@@@@@@@@@@OPERATION COMPLETE Mon Aug 23 10:22:01 PDT 2010 @@@@@@@@@@@@@@@@
@@@@@@@@@@@NEW SEARCH INITIATED AT Mon Aug 23 19:23:09 PDT 2010@@@@@@@@@@@@@@
Revision 506 Mon Aug 16 22:00:36 PDT 2010
PERFORMING MAINTENANCE ROUTINE
SEARCHING: www.TheTvDb.com SHOW NAME: Top Gear EPISODE:
FILE NAME: /var/lib/mythtv/recordings/1162_20100823180000.mpg
SEARCH FOUND:Top Gear ID#: 74608
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear
MATCH!!!! Zap2itID:Top Gear ID.74608
SEARCH FOUND:Top Gear classic ID#: 142961
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear classic
SEARCH FOUND:Top Gear Russia ID#: 87701
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear Russia
SEARCH FOUND:Top Gear Australia ID#: 83234
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear Australia
SEARCH FOUND:Top Gear Xtra ID#: 83407
LOCAL DATABASE UPDATED:/home/mythtv/.mythicalLibrarian/Top Gear Xtra
DEFINED ABSOLOUTE EPISODE NUMBER:
WARNING:SHOW WAS CONSIDERED AN EPISODE BUT INFORMAITON WAS NOT OBTAINABLE
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
HANDLING EPISODE AS GENERAL SHOWING AND DUMPING INTO --doover USER JOB
GENERIC GUIDE DATA WAS SUPPLIED TYPE: Series With Episode Data- Top Gear,
WARNING:%%%%%%%%%%PROGRAM GUIDE DATA IS NOT COMPLETE%%%%%%%%%%%%%%%%%%%%
MOVING FILE: '/var/lib/mythtv/recordings/1162_20100823180000.mpg' to '/var/lib/mythtv/recordings/Showings/Top Gear/Top Gear S0E0 ( Recorded [email protected] on 1162).mpg'
@@@@@@@@@@@@@OPERATION COMPLETE Mon Aug 23 19:23:20 PDT 2010 @@@@@@@@@@@@@@@@
Thanks!

/Greg

Ok, I increased the logging level and took care of a few help messages as well. Please run mythicalLibrarian --update, then mythicalLibrarian --doover and post up a debug log. That should give the information required. From what I'm seeing here it is saying that there is not enough information to work with. I'd really like to see the debug log to find out why it would say that.


- GregoryK - 2010-08-26

Adam:

Here's the debug from output.log after the --doover run just now.

http://pastebin.com/FcSBBwGt

Hope it helps!

/Greg


- outleradam - 2010-08-26

mythicalLibrarian found a match based on the Zap2it ID provided by your guide data and associated it with series 74608. The title match is proper but it is not able to obtain an episode match.

When I get some time I will check that TheTvDb has an episode under that series id on 2010-08-23. It is likely that the shows have not yet been entered into theTvDb and therefore it is impossible for the season and episode to be matched. mythicalLibrarian defaults it into S0E0 (the universal "put me in the library without any other information given" key).

I was able to check theTvDb on my phone and they do not have that episode in it yet. The logs were correct in reporting not sufficient data. Please consider helping out and adding to theTvDb. Season/Episode matches are not possible without data to match.


- tom_a_sparks - 2010-08-27

outleradam Wrote:That is normal. You are using the use original dir option. The dir is created by the location of which mythicalLibrarian is called.

that option has been turned off see http://forum.xbmc.org/showpost.php?p=591445&postcount=486