mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings

outleradam · 2010-01-14 15:06

malte Wrote:Maybe you want to add this as a configurable option.

I was considering that, but then it needs a backup if it gets no information, then, some of my friends in Germany who watch French and Portugeese and Turkish Television would have me create a better system. I guess a Default language will work

Quote:Does it mean you don't need any entries from an online source and just "copy" the information from mythtv database to xbmc? That's exactly what I'm looking for! Please tell me if you will implement this in the next days/weeks. Maybe I will try it by myself but I am not very skilled at bash scrptiing.

One more feature I will try to implement myself (its very special so I don't think anyone else will use it, but who knows...):
I can't rely on all this movie or tv show markers. I don't have "MV" or "SH" entries in my database or something else. But I know it when scheduling my recordings. So I will use different storagegroups for movies and tv shows to have them stored in different directories. In your script I will parse the %DIR% parameter to decide if it is dealing with a movie or tv show. Maybe not the best approach but I am sure it will work.

regards,
malte

please post up some of your guide data descriptors. If you're not using zap2it who are you using?

here is the code for the NFO file generator I've been working on... however, it is not complete and it does not do anything for XBMC yet as XBMC only supports movies and episodes.

Code:

#Create an NFO file based upon extracted information

 GenNFO () {

 echo "<episodedetails>">$mythicalLibrarian/NFOFILE.NFO

 echo "        <title>$ShowName</title>">>"$mythicalLibrarian"/NFOFILE.NFO

 test ! -z $seriesid && echo "        <episodeguide>http://www.thetvdb.com/api/$APIkey/series/$seriesid/all/en.zip<episodeguide>">>"$mythicalLibrarian"/NFOFILE.NFO

 echo "        <rating>$rating</rating>">>"$mythicalLibrarian"/NFOFILE.NFO

 echo "        <plot>$plot</plot>">>"$mythicalLibrarian"/NFOFILE.NFO

 echo "        <genre>$ShowCategory</genre>">>"$mythicalLibrarian"/NFOFILE.NFO

 test ! -z $OriginalAirDate && echo "        <aired>$OriginalAirDate</aired>">>"$mythicalLibrarian"/NFOFILE.NFO

 echo "        <recordstart>$ShowStartTime</recordstart>">>"$mythicalLibrarian"/NFOFILE.NFO

 echo "</episodedetails>">>"$mythicalLibrarian"/NFOFILE.NFO

 }

It will create an NFO file, but it does not do anything for XBMC because XBMC only supports movies and episodes.

outleradam · 2010-01-14 16:16

Please post debug logs from a movie, a episode and a generic episode without subtitle or news/sports und kennt du dine EPG GmbH oder?

malte · 2010-01-14 18:19

Quote:please post up some of your guide data descriptors. If you're not using zap2it who are you using?

First I tried with the original EPG data that comes with my DVB-T. But that was missing complete channels and also subtitles for most of the tv shows. Now I am using a scraper that will read the data from a german tv magazine. What do you mean with guide data descriptors?

Quote:Please post debug logs from a movie, a episode and a generic episode without subtitle or news/sports und kennt du dine EPG GmbH oder?

I will do that when I'm back home. You need it with the original version of your script or with my "de"-modifications?

outleradam · 2010-01-14 20:04

What I really need is the database entries. The mythconverg database contains massive amounts of indormation which is required to properly identify movies and distinguish episodes with data from episodes without data.

The way this is accomplished now is to check the zaptoit/schedules direct program ID. If the ID starts with mv it is a movie. Ep is episode. Sh is show with no episode information. This can be found in mythtv and iTivo txt files. I need to see the debug logs to build the program with an option to gain info about your recordings. MythicalLibrarian must have mysql access set up properly. A debug log should contain all the information I need to support DE data.

malte · 2010-01-14 22:25

Ok, here are some database values. I already checked that earlier. I don't have any of these information in my database (programid, orig airdate, syndicatedepisodenumber, ...). I did not find the txt files you mentioned. I copy and pasted some values from database directly (I hope you can read it):

Table "recorded"

Code:

'title', 'subtitle', 'category', 'recordid', 'seriesid', 'programid', 'originalairdate'

'Tagesschau', '', 'film', 13, '212613461', 'MV212613461', '0000-00-00'

'Tatort: Acht Jahre später', 'Fernsehfilm Deutschland 1974', 'Original Language', 15, '', '', '0000-00-00'

'Scrubs - Die Anfänger', 'Mein scharfes Kindermädchen', 'serie', 11, '60697138', '', '0000-00-00'

Table "recordedprogram"

Code:

'title', 'subtitle', 'category', 'category_type', 'airdate', 'seriesid', 'originalairdate', 'showtype', 'syndicatedepisodenumber', 'programid', 'manualid', 'generic', 'listingsource'

'Tagesschau', '', 'film', 'movie', '2010', '212613461', , '', '', 'MV212613461', 0, 0, 4

'Tatort: Acht Jahre später', 'Fernsehfilm Deutschland 1974', 'Original Language', 'tvshow', '0000', '', , '', '', '', 0, 0, 1

'Scrubs - Die Anfänger', 'Mein scharfes Kindermädchen', 'serie', '', '2006', '60697138', , '', '', '', 0, 0, 4

The interesting one is the first entry "Tagesschau". It's just a 15 minutes news show but it's the only entry in my whole database with a programid and marked as "movie". I record this every day and it's always the same behaviour. So if you manage to detect this correctly you are my man;-)

As already said I am using a scraper that gets my epg data from a tv magazine online service. I searched some german forums before I set this up and it seems to be used quite often. I am satisfied with this data if I could manage to import it into XBMC database (if there is no entry at tvdb or somewhere else).

I did not attach any debug logs because I don't think there is a chance to fill the gaps in this epg data. Please tell me if you want to have a look at the logs or if you need more information from database.

Quote:It will create an NFO file, but it does not do anything for XBMC because XBMC only supports movies and episodes.

What do you mean with this? I thought XBMC would use information from nfo files if provided and that your script is intended to handle movies and episodes.

outleradam · 2010-01-14 23:52

Again, I need the debug logs. I need to see what is being picked up and how it is being picked up.

The whole idea of using an NFO file is for shows with no episode or movie information. Handling for episodes and movies is down pact. The only problem is handling unscrapable show. The idea of the NFO file is that it can get information into the XBMC database when it is not possible by other means. Right now, unscrapable shows are non-library shows. What I'm waiting for is the next version of XBMC so I can add that information into XBMC for things like news/sports, episodes with no information, documentaries.

Ok, it looks like your guide data is a bit different then mine. I need more information about the log. Do you get comskip data generated? Do you have coloumn "airdate"? Is that all the information in the program guide?

I need specific information here. The DE modification will work well. What is the german TV magazine. Where can I find them online? If you would like to speed this along, tell me what will be the differences between generic episodes, episodes with scrapable information, and movies. Also, get me those debug logs so I can see what needs to be changed and what does not. I will have to rewrite the database/database processing sections to extrapolate needed information from your database. Basically, I have to gather enough information from your EPG to not only identify the show, but to identify if the show can be identifed and the type.

malte · 2010-01-15 10:15

Of course, I will provide you all data you need to improve your script. But I really think you should not dig too deep into this. My lack of epg data is my problem not yours. Have a look at my logs and an extract of my epg data. but I think there's no chance to get along with it.

Here's the log of a movie that is not recognized as a movie:

Code:

@@@@@@@@@@@NEW SEARCH INITIATED AT Fri Jan 15 02:17:54 CET 2010@@@@@@@@@@@@@

PERFORMING MAINTENANCE ROUTINE

SEARCH FOUND: ID#:

series was not found the tvdb or this is a movie may be down try renaming 5 x 2 - Fünf mal zwei

DB ENTIRES- RECSTART:2010-01-15 00:50:00- MOVIE:2004- ORIGAIRDATE:- CHID:1002- CAT:spielfilm-

###################DEBUG MODE ENABLED####################

LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.

INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/lom/Dokumente/XBMC/Tools/MythicalLibrarian-

MOVE DIR:/home/lom/Videos/Serien- USING SHOWNAME AS FOLDER:Enabled-

FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/lom- ALTERNATE MOVE DIR:/home/lom/Videos/Serien-

USE ORIGINAL DIR:Disabled NOTIFICATIONS:Disabled DEBUG MODE:Enabled-

INPUT SHOW NAME:5 x 2 - Fünf mal zwei- LOCAL SHOW NAME TRANSLATION:- SENT TVDB SHOW NAME:-

RESOLVED SERIES ID:- RESOVED SHOW NAME:-

INPUT EPISODE NAME:- ABSOLOUTE EPISODE NUMBER:- RESOLVED EPISODE NAME:-

SEASON:- EPISODE:- SYMLINK MODE:MOVE- FILESIZE: 1907188'kB'

FLAGS: MOVEDIR:1- FREE:1226287148'kB'- ALTERNATEMOVEDIR:1- FREE:1226287148'kB'-

PRIMARYMOVIEDIRWRITABLE:1- FREE:1226287148'kB'- ALTERNATEMOVIEDIR:1- FREE:1226287148'kB'-

*DATABASE* RECSTART:2010-01-15 00:50:00- MOVIE YEAR:2004- ORIGINAL SERIES DATE:-

ProgramID:- CHANNEL ID:1002- CATEGORY:spielfilm-

EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:- STARS:0 RATING:1

PLOT: Kurzkritik: Brillant gespieltThrill ** / Anspruch * / Erotik * / Allgemein *** / tv14-Allgemein **** / tv14-Anspruch * / tv14-Thrill ** / tv14-Kamera ** / tv14-Musik * / tv14-Story * / tv14-Erotik * / tv14-GefühlBitter: Rückblenden zeigen den unaufhaltsamen Verfall einer einst großen Liebe. Von François Ozon ("Swimming Pool").Szenen einer gescheiterten Ehe - das Besondere an der tragischen Lovestory zwischen Valeria Bruni-Tedeschi und Stéphane Freiss : François Ozon ("8 Frauen") erzählt sie rückwärts - geschieden, verheiratet, verlobt, verliebt.Info: Die Geschichte des Paares wird rückwärts erzählt - von der Scheidung zum Verlieben. Ähnlich verfährt der Psycho-Thriller "Memento" (2000).

##############LISTING FOLDER PERMISSIONS#################

ORIGIONAL FILE>ls -l /home/lom/Videos/Aufnahmen/Filme/1002_20100115005000.mpg

-rw-r--r-- 1 mythbackend video 1952961108 Jan 15 02:17 /home/lom/Videos/Aufnahmen/Filme/1002_20100115005000.mpg

mythicalLibrarian WORKING DIR>lsmod -l /home/lom/Dokumente/XBMC/Tools/MythicalLibrarian//

total 308

drwxrwxr-x 2 lom         video   4096 Jan 14 11:36 DailyReport

drwxrwxr-x 2 lom         video   4096 Jan  7 19:30 Die Simpsons

drwxr-xr-x 2 mythbackend video   4096 Jan 12 22:16 Dr. House

drwxrwxr-x 2 lom         video   4096 Jan  6 20:04 King of Queens

drwxrwxr-x 2 lom         video   4096 Jan  6 14:44 Scrubs

drwxrwxr-x 2 lom         video   4096 Jan  9 16:47 Tagesschau

drwxrwxr-x 2 lom         video   4096 Jan  9 16:46 Tatort

drwxr-xr-x 2 lom         users   4096 Jan 11 08:47 Tatort:Klassentreffen

-rw-r--r-- 1 mythbackend video    660 Jan 15 02:17 comskiplog.tracking

-rw-rw-rw- 1 lom         users    945 Jan 14 00:21 doover.sh

-rw-rw-r-- 1 lom         video    749 Jan 11 08:48 doover.sh~

-rw-r--r-- 1 mythbackend video     50 Jan 15 02:17 markupframes.txt

-rw-rw-r-- 1 lom         video      1 Jan 15 02:17 markupstart.txt

-rw-rw-r-- 1 lom         video      1 Jan 15 02:17 markupstop.txt

-rwxrwxr-x 1 lom         video  46294 Jan 14 19:56 mythicalLibrarian.sh

-rw-rw-r-- 1 lom         video  46294 Jan 14 19:55 mythicalLibrarian.sh~

-rw-rw-r-- 1 lom         video 139286 Jan 15 02:17 output.log

-rw-rw-r-- 1 lom         video     37 Jan 14 18:41 shn.txt

-rw-rw-r-- 1 lom         video     12 Jan 14 18:41 sid.txt

-rw-rw-r-- 1 lom         video   3116 Jan 14 18:41 working.xml

####################END OF DEBUG LOG#####################

MOVING FILE: /home/lom/Videos/Aufnahmen/Filme/1002_20100115005000.mpg to /home/lom/Videos/Filme// (2004).mpg

@@@@@@@@@@@@@OPERATION COMPLETE Fri Jan 15 02:17:56 CET 2010 @@@@@@@@@@@@@@@@

In this case you had a chance to recognize it because it used category = 'spielfilm' (that means movie in german). But that's only reliable in one direction: I did not find a tvshow marked as "spielfilm" but a lot of movies that are marked with category = 'Original language' and category_type = 'tvshow'. And there are a lot of tvshows also marked as 'Original language'.

Another log with 'Original Language' as category (again, that`s also used for tvshows, news and anything else)

Code:

@@@@@@@@@@@NEW SEARCH INITIATED AT Thu Jan 14 00:21:28 CET 2010@@@@@@@@@@@@@

PERFORMING MAINTENANCE ROUTINE

SEARCHING: www.TheTvDb.com SHOW NAME: Tatort: Acht Jahre später EPISODE: Fernsehfilm Deutschland 1974

FILE NAME: /home/lom/Videos/Aufnahmen/Tatort/1260_20100113224900.mpg

SEARCH FOUND: ID#:

series was not found the tvdb or this is a movie may be down try renaming Tatort: Acht Jahre später

DB ENTIRES- RECSTART:2010-01-13 22:49:00- MOVIE:- ORIGAIRDATE:- CHID:1260- CAT:Original Language-

###################DEBUG MODE ENABLED####################

LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.

INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/lom/Dokumente/XBMC/Tools/MythicalLibrarian-

MOVE DIR:/home/lom/Videos/Serien- USING SHOWNAME AS FOLDER:Enabled-

FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/lom- ALTERNATE MOVE DIR:/home/lom/Videos/Serien-

USE ORIGINAL DIR:Disabled NOTIFICATIONS:Disabled DEBUG MODE:Enabled-

INPUT SHOW NAME:Tatort: Acht Jahre später- LOCAL SHOW NAME TRANSLATION:- SENT TVDB SHOW NAME:Tatort:%20Acht%20Jahre%20später-

RESOLVED SERIES ID:- RESOVED SHOW NAME:-

INPUT EPISODE NAME:Fernsehfilm Deutschland 1974- ABSOLOUTE EPISODE NUMBER:- RESOLVED EPISODE NAME:Fernsehfilm Deutschland 1974-

SEASON:- EPISODE:- SYMLINK MODE:MOVE- FILESIZE: 2148660'kB'

FLAGS: MOVEDIR:1- FREE:1237654968'kB'- ALTERNATEMOVEDIR:1- FREE:1237654968'kB'-

PRIMARYMOVIEDIRWRITABLE:1- FREE:1237654968'kB'- ALTERNATEMOVIEDIR:1- FREE:1237654968'kB'-

*DATABASE* RECSTART:2010-01-13 22:49:00- MOVIE YEAR:- ORIGINAL SERIES DATE:-

ProgramID:- CHANNEL ID:1260- CATEGORY:Original Language-

EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:- STARS:0 RATING:1

PLOT: Kommissar Haferkamp ist in Gefahr. Der Einbrecher Brossberg wird nach 8 Jahren Haft entlassen. Haferkamp hatte ihn gestellt. Und bei einem Feuergefecht, das der Verhaftung vorausging, hat Haferkamp Brossbergs Bruder getötet.

##############LISTING FOLDER PERMISSIONS#################

ORIGIONAL FILE>ls -l /home/lom/Videos/Aufnahmen/Tatort/1260_20100113224900.mpg

-rw-r--r-- 1 mythbackend video 2200228108 Jan 14 00:21 /home/lom/Videos/Aufnahmen/Tatort/1260_20100113224900.mpg

mythicalLibrarian WORKING DIR>lsmod -l /home/lom/Dokumente/XBMC/Tools/MythicalLibrarian//

total 284

drwxrwxr-x 2 lom         video   4096 Jan 13 10:56 DailyReport

drwxrwxr-x 2 lom         video   4096 Jan  7 19:30 Die Simpsons

drwxr-xr-x 2 mythbackend video   4096 Jan 12 22:16 Dr. House

drwxrwxr-x 2 lom         video   4096 Jan  6 20:04 King of Queens

drwxrwxr-x 2 lom         video   4096 Jan  6 14:44 Scrubs

drwxrwxr-x 2 lom         video   4096 Jan  9 16:47 Tagesschau

drwxrwxr-x 2 lom         video   4096 Jan  9 16:46 Tatort

drwxr-xr-x 2 lom         users   4096 Jan 11 08:47 Tatort:Klassentreffen

-rw-r--r-- 1 mythbackend video    660 Jan 14 00:21 comskiplog.tracking

-rw-rw-rw- 1 lom         users    755 Jan 13 11:28 doover.sh

-rw-rw-r-- 1 lom         video    749 Jan 11 08:48 doover.sh~

-rw-r--r-- 1 mythbackend video     50 Jan 14 00:21 markupframes.txt

-rw-rw-r-- 1 lom         video      1 Jan 14 00:21 markupstart.txt

-rw-rw-r-- 1 lom         video      1 Jan 14 00:21 markupstop.txt

-rwxrwxr-x 1 lom         video  46287 Jan 12 08:38 mythicalLibrarian.sh

-rw-rw-r-- 1 lom         video  46205 Jan 12 08:37 mythicalLibrarian.sh~

-rw-rw-r-- 1 lom         video 123028 Jan 14 00:21 output.log

-rw-rw-r-- 1 lom         video      0 Jan 14 00:21 shn.txt

-rw-rw-r-- 1 lom         video      0 Jan 14 00:21 sid.txt

-rw-rw-r-- 1 lom         video     54 Jan 14 00:21 working.xml

####################END OF DEBUG LOG#####################

malte · 2010-01-15 10:19

The data I posted yesterday was an extract of my mythconverg database. I only pasted some columns because most information like description, starttime, endtime, ... will be useless in this case. But here is a little extract from my original epg data file that is used with the mythfilldatabase to get the data into mythconverg:

Code:

<programme start="20100113225000" stop="20100114002000" channel="wdr">

        <title lang="de">Tatort</title>        

        <sub-title lang="de">Acht Jahre sp&#xE4;ter</sub-title>        

        <category lang="de">Spielfilm</category>

        <genre lang="de">Krimi</genre>

        <date lang="de">1973</date>

        <video>

        <aspect>4:3</aspect>

        </video>

        <credits>

        <director>Wolfgang Becker</director>

        <actor>Max Mairich (Direktor)</actor>

        ...

        </credits>        

        <desc lang="de">Kurzkritik: ...</desc>    

</programme>

<programme start="20100115200000" stop="20100115201500" channel="wdr">

        <title lang="de">Tagesschau</title>        

        <category lang="de">Film</category>

        <genre lang="de">Nachrichten</genre>

        <date lang="de">2010</date>

        <audio>

        <stereo>Stereo</stereo>

        </audio>

        <video>

        <aspect>4:3</aspect>

        </video>

    </programme>

First one is marked as "Spielfilm" (movie) in my epg but will be stored as "Original Language" in mythconverg. Second one is a news show that is marked "Film" (short word for movie) and marked as "movie" in mythconverg.

Here's an extract of mythconverg program table (all columns now. sorry, if it is unreadable):

Code:

'chanid', 'starttime', 'endtime', 'title', 'subtitle', 'description', 'category', 'category_type', 'airdate', 'stars', 'previouslyshown', 'title_pronounce', 'stereo', 'subtitled', 'hdtv', 'closecaptioned', 'partnumber', 'parttotal', 'seriesid', 'originalairdate', 'showtype', 'colorcode', 'syndicatedepisodenumber', 'programid', 'manualid', 'generic', 'listingsource', 'first', 'last', 'audioprop', 'subtitletypes', 'videoprop'

1259, '2010-01-13 22:50:00', '2010-01-14 00:20:00', 'Tatort: Acht Jahre später', '', 'Fernsehfilm Deutschland 1974', 'Original Language', 'tvshow', '0000', 0, 0, '', 0, 0, 0, 0, 0, 0, '', , '', '', '', '', 0, 0, 1, 1, 1, '', '', ''

1515, '2010-01-11 20:00:00', '2010-01-11 20:15:00', 'Tagesschau', '', '', 'film', 'movie', '2010', 0, 0, '', 0, 0, 0, 0, 0, 0, '212613461', , '', '', '', 'MV212613461', 0, 0, 4, 0, 0, '', '', ''

Again, I don't want you to fix my epg problems;-). I think I could manage this with different storage groups in mythtv and then have a little branch in your script that will ask for the directory name and take this to decide if it is a movie, a tvshow or whatever. I don't expect you to put something like this in your script because it seems to be a very special problem. So I will hijack this in my local copy.

Quote:The whole idea of using an NFO file is for shows with no episode or movie information

That's exactly what I was talking about. I want to import information like title, subtitle, description, actors, ... that is already stored in my epg data into XBMC database if the scraper can't find it at his online source. I tried this with some of my unscrapable shows and it seems to work as I expect it. I only have to provide a unique Series and Episode number that I am sure of that it's not existing for another episode. I tested something like S01E10000 and it was imported without any problem. Thank you very much for this usefull information! I did not know about nfo files before...

outleradam · 2010-01-15 15:31

Can you find the movie year in categories other then movies?

Here is what I am thinking as a soloution for your guide data. Tell me what you think. I am going to restructure the database section to perform processing for different types of guide data suppliers. Also cleaning up the code so databases like iTivo can be added.

If movie year is not 0000 or "" or category is film then it is a movie?

If movie year = 0000 or "" and there is a subtitle Or a original airdate then it is a series?

If none of the above then it is a show with no data

malte · 2010-01-15 21:04

Good idea, but that isn't reliable in my case. I have a lot of movies without airdate and a lot of tvshows that have one.

Thank you for digging into this but I think I have to do this storagegroup thing. I will tell you if this works. And thanks again for the hint with the nfo and the code to do this. I will try to get this working if there aren't any entries at tvdb.