XBMC Community Forum
mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings - Printable Version

+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for XBMC (/forumdisplay.php?fid=116)
+--- Thread: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings (/showthread.php?tid=65644)



- outleradam - 2011-01-05 18:57

I will be releasing a new stable version tonight... This uses a new database processor which increases speed and accuracy of recognition from about 30 seconds from start to finish, down to about 3 seconds. When upgrading, it is important that you use
Code:
mythicalLibrarian --update
mythicalLibrarian --doMaintenance
Otherwise your episodes will be recognized wrong until theTvDb updates their entries. For some shows this may be never.


- colbert - 2011-01-06 01:56

Okay, I tried to set this up and I definitely didn't do it right lol. I followed instructions here from wiki:

Code:
mkdir ~/.mythicalLibrarian && mkdir ~/.mythicalLibrarian/mythicalSetup && cd ~/.mythicalLibrarian/mythicalSetup
curl http://mythicallibrarian.googlecode.com/svn/trunk/mythicalSetup.sh>./mythicalSetup.sh
sudo chmod +x ./mythicalSetup.sh && sudo ./mythicalSetup.sh

I went through the steps but I wasn't clear on all of them, especially the IP:PORT of XBMC. My setup is that I am running XBMC from a desktop box, and have Myth running separately on a backend server. This desktop is a frontend and I'd like to replace mythfrontend with XBMC.

My desktop is .101 and server .100.

Here is output.log:

Code:
@@@@@@@@@@@NEW SEARCH INITIATED AT Wed Jan  5 18:46:16 EST 2011@@@@@@@@@@@@@@
Revision 909 Wed Jan 5 18:39:04 EST 2011 on Linux
%%%NO DATABASE INFORMATION. CHECK LOGIN/PASS OR FILE %%%%%
SEARCHING: www.TheTvDb.com SHOW NAME: mister rogers EPISODE: show 1332
DATE:  FILE NAME: testfile.ext
FOUND:Mister Rogers' Neighborhood ID#:77750 WITH CONFIDENCE:-2
Creating MythicalLibrarian Database Folder
AIRDATE FAILED. TITLE MATCH ON ABSOLUTE EPISODE NUMBER:
###################DEBUG MODE ENABLED####################
MY USER NAME:bobby-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/bobby/.mythicalLibrarian-
MOVE DIR:/home/bobby/.mythicalLibrarian/mythicalSetup/Episodes-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/bobby/.mythicalLibrarian/mythicalSetup/Episodes-
USE ORIGINAL DIR:Enabled NOTIFICATIONS:Enabled
INPUT SHOW NAME:mister rogers- LOCAL SHOW NAME TRANSLATION:- ShowName:mister rogers
DATABASE UPDATED:- TVDB LAST UPDATED:- CURRENT:-
RESOLVED SERIES ID:77750- RESOVED SHOW NAME:Mister Rogers' Neighborhood-
INPUT EPISODE NAME:show 1332- Absolute EPISODE NUMBER:462- RESOLVED EPISODE NAME:Show 1332-
SEASON:S07- EPISODE:E07- SYMLINK MODE:MOVE- FILESIZE: 0kB-
CONFIDENCE:-3 LOGIC PATH:  Attempted Match On Zap2itID: Failed; Attemted match on Standard Logic: Failed; Attemted Title match on Fuzzy Logic: Sucessful; Attemped match on OriginalAirDate: Failed; Attemted SubTitle match on Standard logic: Sucessful;
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:1558480kB- WORKINGDIR:1 FREE:1558480kB-
MOVEDIRWRITABLE:1- FREE:1558480kB- ALTERNATEMOVEDIR:1- FREE:1558480kB-
PRIMARYSHOWDIRWRITABLE:1-  FREE:1558480kB-ALTERNATESHOWDIRWRITABLE:1- FREE:1558480kB-
PRIMARYMOVIEDIRWRITABLE:1- FREE:1558480kB- ALTERNATEMOVIEDIR:1- FREE:1558480kB-
DATABASE TYPE:-
RECSTART:- MOVIE YEAR:- ORIGINAL SERIES DATE:-
PROGRAMID:- CHANNEL ID:- CATEGORY:- GOFORDOOVER:0-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:- STARS: RATING:0
ZAP2IT SERIES ID:- MATCHED TVDB SERIES ID:-
PLOT:
####################END OF DEBUG LOG#####################
CREATING FOLDER: /home/bobby/.mythicalLibrarian/mythicalSetup/Episodes/Mister Rogers Neighborhood

Appreciate any help, thanks Smile


- outleradam - 2011-01-06 04:11

You need to install this on 100

on mythbackend type:
Code:
ctrl+alt+F1
login as privilaged user
Code:
sudo apt-get install ssh
Then from the other computer type this:
Code:
ssh *yourLogin*@192.168.1.100
mkdir ~/.mythicalLibrarian && mkdir ~/.mythicalLibrarian/mythicalSetup && cd ~/.mythicalLibrarian/mythicalSetup
curl http://mythicallibrarian.googlecode.com/svn/trunk/mythicalSetup.sh>./mythicalSetup.sh
sudo chmod +x ./mythicalSetup.sh && sudo ./mythicalSetup.sh
The first time you need to login as privilaged user. mythcialSetup will take care of setting up privilages on the user mythtv if user mythtv does not have sudo privilages.

After you've run mythicalSetup and you have permissions established for mythtv,
Code:
sudo su mythtv
sudo mythicalLibrararian --update
as user mythtv. The XBMC IP will be 192.168.1.100:8080 and make sure you have the control from web server option enabled.


- colbert - 2011-01-06 04:21

^^^ Okay I am going to do this, however the .100 does not have XBMC, only mythtv. I have XBMC only on .101. Still run mythicalLibrarian on .100?

Thanks


- outleradam - 2011-01-06 16:26

mythicalLibrarian works from the mythtv side to provide a proper naming convention for XBMC to recognize. It takes a name like 1024_201101011200.mpg (chan_YearMonthDayTime.ext), references the mythtv database, changes the name to Title/Season X/Title SxxExx (Subtitle).ext. It then notifies XBMC that there were changes and to rescan the library. You must have your source set in XBMC in order for the process to work. So you share the files with SAMBA (SMB://) or any other supported method and set the source in XBMC as TV Shows or Movies and mythicalLibrarian manages the XBMC library automatically. Every once in a while you log into the MythTV machine and type mythicalLibrarian --doover to rescan unrecognized shows.


- GregoryK - 2011-01-06 19:58

Adam:

Still working here on getting my library transcoded so that mythicalLibrarian and XBMC don't result in bad commskips. With the help of my local guru, I'm going to run a script that does the following to each of my recordings:

Code:
echo Processing 1022_20091230200000.mpg
mythcommflag -c 1022 -s 2009-12-30-20-00-00 --gencutlist
mythtranscode --chanid 1022 --starttime 2009-12-30-20-00-00 --mpeg2 --honorcutlist
rm /mnt/video2/store/1022_20091230200000.mpg
mv /mnt/video2/store/1022_20091230200000.mpg.tmp /mnt/video2/store/1022_20091230200000.mpg
mythcommflag --file 1022_20091230200000.mpg --rebuild
NEWFILESIZE=`du -b /mnt/video2/store/1022_20091230200000.mpg | cut -f1`
echo "UPDATE recorded SET basename='1022_20091230200000.mpg',filesize='$NEWFILESIZE',transcoded='1' WHERE chanid='1022' AND starttime='2009-12-30-20-00-00'" > update-database.sql
mysql --user=dbuserHere --password=password mythconverg < update-database.sql
rm /mnt/video2/store/1022_20091230200000.mpg*.png

After this, per your instructions, I was going to mythicalLibrarian --scan mpg all of the recordings in my storage directories. My hope is that the transcoded files will have the uniform frame rate that XBMC assumes.

Can you see any problems with this approach? (I realize this is sort of off-topic for this thread, but I thought it might be relevant if others are having problems with playing their recordings on XBMC.)

Thanks as always,

/Greg


- outleradam - 2011-01-06 21:05

Code:
read -p "Enter folder to scan:" Folder
read -p "Enter DBUserName:" DBUsername
read -p "Enter DBPassWord:" DBPassword

echo "creating list of files in $Folder"
ls -1  "$Folder" > ./tempList

while read line
do
  test "${line:0:3}" != "mpg" && continue
  FileName=`basename "$line"`
  echo Processing "$FileName".mpg

#This part can be automated... it requires a bit of reference using

#Chanid=`mysql -u"$DBUserName" -p"$DBPassWord" -e \'use mythconverg; select chanid from recorded where basename like "$FileName";\'`|sed -n 2p
#Starttime=`mysql -u"$DBUserName" -p"$DBPassWord" -e \'use mythconverg; select starttime from recorded where basename like "$FileName";\'|sed -n 2p
  mythcommflag -c $Chanid -s $Starttime --gencutlist
  mythtranscode --chanid $Chanid --starttime $Starttime --mpeg2 --honorcutlist
  rm "$Folder"/"$FileName".mpg
  mv "$Folder"/"$FileName.mpg".tmp "$Folder"/"$FileName".mpg  mythcommflag --file "$FileName".mpg --rebuild
  NEWFILESIZE=`du -b "$Folder"/"$FileName".mpg | cut -f1`
  echo "UPDATE recorded SET
#Same for starttime
basename='$FileName.mpg',filesize='$NEWFILESIZE',transcoded='1' WHERE chanid=\'$Chanid\' AND starttime=\'$Starttime\'" > update-database.sql
  mysql --user=$DBUserName --password=$DBPassWord mythconverg < update-database.sql
  rm "$Folder"/"$FileName".mpg*.png
done < ./tempList
echo "Removing previously created list of files in $Folder"
rm ./tempList

I'm not near a linux computer right now, so I can't test this out, but it should work with a bit of tweaking..

I don't really see what you're doing with all of this though.


- GregoryK - 2011-01-06 22:12

outleradam Wrote:I don't really see what you're doing with all of this though.

Good question. Back a few pages, I asked if you had any idea why the playback in XBMC was skipping commercials in the wrong places. You explained that it was due to XBMC assuming a constant framerate, and gave me a workaround:

outleradam Wrote:GregoryK: The problem is XBMC's handling of MPEG2-TS variable framerate. <<snip>>

The patch is to have MythTV transcode every file. This creates a fixed framerate for every show. This should be done before mythicalLibrarian runs. So, set MythTV to run only 1 job at a time. Set MythTV to do MythTranscode job, select MythCommFlag checkbox, and have mythicalLibrarian run last.

Because I have a bunch of already-mythicalLibrarian-processed recordings in my library, you later suggested that I --undo those, then transcode them using myth, then run mythicalLibrarian --scan to get the transcoded files into my XBMC library.

Thus, the script is designed to automate the process of transcoding my existing mythtv recordings. My hope is that, per your suggested "patch" comment quoted above, that this will let me watch my recordings in XBMC with the commercials skipped appropriately.

...am I making any sense? or have I misunderstood your earlier suggestions and gone off in the wrong direction?


- colbert - 2011-01-07 00:31

outleradam Wrote:mythicalLibrarian works from the mythtv side to provide a proper naming convention for XBMC to recognize. It takes a name like 1024_201101011200.mpg (chan_YearMonthDayTime.ext), references the mythtv database, changes the name to Title/Season X/Title SxxExx (Subtitle).ext. It then notifies XBMC that there were changes and to rescan the library. You must have your source set in XBMC in order for the process to work. So you share the files with SAMBA (SMB://) or any other supported method and set the source in XBMC as TV Shows or Movies and mythicalLibrarian manages the XBMC library automatically. Every once in a while you log into the MythTV machine and type mythicalLibrarian --doover to rescan unrecognized shows.

Okay, so (apologize for the noobness here but just getting the hang of XBMC after longtime mythfrontend reliance which I'm trying to abandon hehe) I run it from the .100, which is the Myth BE. I currently have the Myth recordings mounted on the .101-- the XBMC box-- as /media/myth, so I add video source /media/myth, and mythicalLibrarian automatically manages the /media/myth source correct? Will it interfere with the rest of my library (all of which are on the .101 XBMC machine)?

Logging into the Myth BE is no prob as I have a constant urxvt screen session open with it and do a lot on it with httpd as it is, so that should work out well.

Thanks very much!


- outleradam - 2011-01-07 16:09

^^ You can have as many sources as you'd like in XBMC. Each one is scanned individually. In fact, mythicalLibrarian creates 3 soures. 1 Movies source, 1 Episode source and 1 Generic Showings source.

Episodes and Generic showings should be handled as "Episodes" by XBMC. This allows XBMC to differentiate between shows it should scrape information and those which it should not. mythicalLibrarian handles the "Generic Showings" directory by placing all known information into a NFO file which, when placed into the same folder as "Episodes", creates conflicts and inability to obtain information.

So, you simply run mythicalLibrian --update on 100(mythtv box), set XBMCIP as 192.168.1.101:8080(xbmc box), set your XBMC sources as above, and let everything operate from there.