mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings

  Thread Rating:
  • 3 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #31
I will need a debug log of that event. This looks like inconstant data where the show was attempted to be scraped as a show and a movie at the same time. That 2006 movie year should not be there and it does not make since since it is an EPIT. That may have screwed with mythicalLibrarian and somehow it was detected at 2006 and there was a $ in there where it shouldn't have been and the internet went down for a minute. I can't really tell why mythicalLibrarian made it's decisions without a debug log to tell what it's thinking. I'm setting mythTV to record curious george. I'm sure my kid will like that.

Replace line 291 with this so you don't have to reset all of your settings:
Code:
test "$TvDbTime" = "" && TvDbTime="1"
That was part of the offline file handling which had a $ after &&
Uploading new version now.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
(This post was last modified: 2010-01-20 07:21 by outleradam.)
find quote
tim- Offline
Senior Member
Posts: 110
Joined: Oct 2009
Reputation: 0
Post: #32
Okay, I updated that line and I'll set some shows to record. I am having this same problem with every show. I have not been able to get mythicalLibrarian working yet. I had some permissions issues with an NFS mount and the mythtv user but that is all working now

I just did a pastebin of a debug log at

http://pastebin.com/m2ffdf439

thanks
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #33
From what I'm seeing, you do not have permission to move files from /var/lib/mythtv/recordings. try changing your record dir to a better location, like /home/mythtv/recordings.

You do not have permission to write to either of your primary move dirs. add yourself to the mythtv group and make sure mythtv is a member of the group which can access your primary dirs.

I will do some more testing when I get on my computer tonight. This may be something to do with permission errors and needs better handling.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
find quote
tim- Offline
Senior Member
Posts: 110
Joined: Oct 2009
Reputation: 0
Post: #34
I can try moving my recordings around. I do have write access to all of the move directories, thats what seemed odd to me
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #35
MythicalLibrarian tests write and delete access thoroughly. It starts by checking free space ( original file size for movedirs, 5mb for working dir, 1 byte for originaldir), then creates a small arbitrary file with about 10 bytes of data, checks that the file was created, then deletes the file and checks that it was deleted. If any part fails then writable flag is set to 0. Otherwise if sucessful writable is set to 1. I may need to look at how the rest of the program works if original dir is 0.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
(This post was last modified: 2010-01-20 20:33 by outleradam.)
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #36
I've tried running this a few different ways and it seems to work well even with no database access and the errors between the guide and thetvdb. It recognizes the show well.
Here is the very first one: http://pastebin.com/m226812fc
The 'no match found' is because I am in database mode with no database to back it up.


Try this.
download this and put it on your desktop: https://sourceforge.net/projects/mythica...h/download
download this and put it on your desktop: https://sourceforge.net/projects/mythicallibrari/files/mythicalLibrarianBeta/librarian-notify-send/download

to set up mythicalLibrarian:

Code:
sudo apt-get install curl agrep libnotify-bin
sudo mkdir /home/mythtv /home/mythtv/mythicalLibrarian /home/mythtv/Episodes /home/mythtv/Movies /home/mythtv/Videos /home/mythtv/FailSafe
sudo chown -R mythtv:mythtv /home/mythtv /home/mythtv/mythicalLibrarian /home/mythtv/Episodes /home/mythtv/Movies /home/mythtv/Videos /home/mythtv/FailSafe
sudo mv ~/Desktop/mythicalLibrarian.sh /home/mythtv/mythicalLibrarian/mythicalLibrarian.sh
sudo mv ~/Desktop/librarian-notify-send /usr/local/bin/librarian-notify-send
sudo chmod +x /usr/local/bin/librarian-notify-send
sudo chmod +x /home/mythtv/mythicalLibrarian/mythicalLibrarian.sh
Then add your username to mythtv group
Code:
sudo adduser Your-User-Name-Here mythtv
See use of mythicalLibrarian on MythTV Applications for setting up mythtv here:http://wiki.xbmc.org/?title=MythicalLibr...plications

I just uploaded a new version which corrects the cat could not find lastupdated.time. This is purely cosmetic. Also, i set the default failsafe dir to a different folder.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
(This post was last modified: 2010-01-21 08:31 by outleradam.)
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #37
I just had a serious branching error. I need a copy of mythicalLibrarian before yesterday. If anyone has one, please get it to my e-mail. I am now in posession of a single copy which is a month old because of stupidity on my part. I edited the one on my media center accidentally and then saved it, uploaded it to sourceforge and tested my instructions in the last post, thereby overwriting every copy I had. So, please, if anyone can get one to me it would be very much appreciated. I hope Tim- still has his and did not overwrite it with a 'new' one. If anyone has a newer copy, pastebin it please.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
(This post was last modified: 2010-01-22 16:05 by outleradam.)
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #38
Anyone have a version from january 16 or newer? I'd really hate to have to recode everything.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #39
I found a copy of mythicalLibrarian on a XBMCfront-end/MythTVback-end-computer-on-a-stick which I had created earlier. What's missing can be rewritten.

https://sourceforge.net/projects/mythicallibrari/files/

Whew, talking about freaking out. That's what happens when I take on too many projects at a time.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
(This post was last modified: 2010-01-23 08:36 by outleradam.)
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #40
Alright, new beta https://sourceforge.net/projects/mythica...h/download

Starting the 1 month clock now.

Substantial changes were made which requires the deletion of all libraries created by mythicalLibrarian. Basically, wipe out the entire ~/mythicalLibrarian/ folder
Keep the following files and folders from your old version (if applicable):
  • dir.tracking
  • comskiplog.tracking
  • doover.sh
  • showtranslations
  • output.log
  • /dailyreport/
Delete:
  • All database folders
  • shn.txt
  • sid.txt
  • markupstart.txt
  • markupstop.txt
  • current.time
  • working.xml

All other files and folders must be deleted or wait until the next mythicalLibrarian established database update cycle from the UpdateDatabase=84000 parameter

Database changes are:
Zap2it database parsing
formatting changes to *.ename.txt
correction to lastupdated.time
addition of *.actualEname.txt

Other substantial changes:
better fuzzy logic recognition of shows
better fuzzy logic recognition of episode names
rebuilt zap2it id matching
better logging of errors


Yet to come:
Cosmetic and bug fixes

Guys- I need bugs to fix. Please use the heck out of this. Record the most screwed up programs. Get your money's worth out of your TV tuner(s) and keep your XBMC library rotating.

My goal is if a recorded episode or movie does not make it from your MythTV backend to your XBMC Library, then it's because your cable provider is sending incomplete/incorrect information, or you have not signed up for an account and added information to TheTvDb.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
(This post was last modified: 2010-01-25 01:24 by outleradam.)
find quote
adecker89 Offline
Junior Member
Posts: 1
Joined: Jan 2010
Reputation: 0
Post: #41
I actually just filed an issue on your googlecode site before I saw the link to here. Seems you can't post a reply there anyway...

I found that df will split its output into multiple lines if your filesystem path is too long for its formatting ( lvm tends to cause this), and this caused the freespace variables to bomb on me:

MoveDirFreeSpace=`df $MoveDir|sed -n 2p|awk '{print $4}'`

was giving me back spaces

But I found a real nice fix for this: "df -P" forces df to print results 1 per line. I added that to the four lines and now everything's running smoothly.
find quote
Bueller Offline
Junior Member
Posts: 8
Joined: Jan 2010
Reputation: 0
Post: #42
I just found another issue. The maintenance routine was not deleting the comskip files, but was removing the entries from comskiplog.tracking. The fix was easy, the rm command didn't have quotes around it.

I changed rm $FileToCheck to rm "$FileToCheck" which fixed it right up.
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #43
adecker89: Thanks. great detective work!
Bueller: Thanks. Change was added.

I really appreciate the help guys! I can't possibly check every configuration on my own. I really appreciate it.

New version here: https://sourceforge.net/projects/mythica...h/download


Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
find quote
okiekyle Offline
Junior Member
Posts: 14
Joined: Jan 2010
Reputation: 0
Post: #44
Thanks so much for all the work you've done on this. This is exactly what I've been looking for. However, I have yet to get it to work successfully for me.

I have the version from yesterday (01/25)

Here is the log from a few recent attempts:

Code:
@@@@@@@@@@@NEW SEARCH INITIATED AT Mon Jan 25 23:12:00 CST 2010@@@@@@@@@@@@@
SEARCHING: www.TheTvDb.com SHOW NAME: Everybody Loves Raymond EPISODE: Young Girl
FILE NAME: /home/mythtv/Movies/1062_20100125224500.mpg
creating home mythicalLibrarian and log file
SEARCH FOUND:Everybody Loves Raymond ID#: 73663
DEFINED ABSOLOUTE EPISODE NUMBER:
%%%%%%%%%%www.TheTvDB.com information is incomplete Everybody Loves Raymond, Young Girl
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%OPERATION FAILED Mon Jan 25 23:12:02 CST 2010 %%%%%%%%%%%%%%%%%
@@@@@@@@@@@NEW SEARCH INITIATED AT Tue Jan 26 00:23:10 CST 2010@@@@@@@@@@@@@
SEARCHING: www.TheTvDb.com SHOW NAME: According to Jim EPISODE: ABCs  123s
FILE NAME: /home/mythtv/Movies/1081_20100125230300.mpg
creating home mythicalLibrarian and log file
SEARCH FOUND:According to Jim ID#: 75926
DEFINED ABSOLOUTE EPISODE NUMBER:
%%%%%%%%%%www.TheTvDB.com information is incomplete According to Jim, ABC's & 123's
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%OPERATION FAILED Tue Jan 26 00:23:12 CST 2010 %%%%%%%%%%%%%%%%%
@@@@@@@@@@@NEW SEARCH INITIATED AT Tue Jan 26 12:21:27 CST 2010@@@@@@@@@@@@@
creating home mythicalLibrarian and log file
SEARCH FOUND:The View ID#:
DEFINED ABSOLOUTE EPISODE NUMBER:
%%%%%%%%%%www.TheTvDB.com information is incomplete The View,
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%OPERATION FAILED Tue Jan 26 12:21:28 CST 2010 %%%%%%%%%%%%%%%%%

On the 1st one (Everybody Loves Raymond - Young Girl) I can't figure out why it didn't find the info. It's there in TheTVDB.com (Season 5, Episode 8). Maybe I'm missing something.

On the 2nd one (According to Jim - ABC's & 123's) I notice that the line where it shows "Searching..." it lists "ABCs 123s" but later when it says "TheTvDB.com is incomplete..." it lists "ABC's & 123's". Note the apostrophes and ampersand are missing in the "Searching..." line. Maybe there's a problem with those characters? On TheTVDB, the episode is listed as "ABC's and 123's" so maybe the difference between "&" and "and" is the problem.

On the 3rd one (The View), the guide didn't have an episode name to it's pretty obvious what the problem was. I was wondering if, when you can't figure out what episode it is (missing or no match), could you go ahead and rename, copy, etc. using something like "ShowName-EpisodeName (from the guide, if any)-AirDateTime". Something like that so I could still find the show in XBMC and there would be as much information as possible in the file name so manual clean-up would be easier. Maybe it should be a configuration option to do this when you can't figure it out from the data you have.

Any other ideas?

Thanks
find quote
outleradam Offline
Donor
Posts: 1,722
Joined: Sep 2009
Reputation: 20
Location: Louisiana
Post: #45
Did you download the SourceForge or the Google Code link? To be able to tell, check at the top of the mythicalLibrarian.sh file. Yesterday, I started labeling creation dates on line 2 of the code.
This is always the most up-to-date version: https://sourceforge.net/projects/mythica...h/download

Google code is what I consider to be the "stable" version, basically a distribution for ease of use where I set everything at defaults and make it "clean". The SourceForge one is my working copy. It's running very well and has not had any problems. I should update the google code version.

I noticed that it seems to be making folders every time for you. You should check your permissions. You can do the "quick install" from this post which should work well http://forum.xbmc.org/showpost.php?p=489...stcount=36


I really need debug logs to do anything. You should set the loging to debug mode. I set it to disabled in the "stable version" because it recognized everything I could throw at it, but there have been a few problems since then and really, to nip all problems from this script I need help from others.

Use mythicalLibrarian to make a library out of your MythTV files. Leave the recording to MythTV and use XBMC as your library.
Installation and Instructions:http://wiki.xbmc.org/index.php?title=MythicalLibrarian
Technical Support:http://forum.xbmc.org/showthread.php?tid=65644
[Image: newsig-1.png]
[url=http://forum.xda-developers.com/showthread.php?tid=1081892][/url]
find quote
Post Reply