Kodi Community Forum
mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings (/showthread.php?tid=65644)



- outleradam - 2010-08-27

Oh. Ok! I see what you are talking about. It is a problem with mythicalSetup making extra dirs which go unused. I'll take care of that tonight.


- outleradam - 2010-08-29

SVN is having technical difficulties right now. It will be Online, but I would advise against building on SVN for now. Please use the stable option in mythicalSetup until I get it all ironed out. It will be up soon. Just be patient and revert to stable until further notice.

I'm working on: permissions, recognition and folder tracking. It could get ugly if you grab a SVN at the wrong time. Changes are coming fast and tested nearly immediately then revised.

This may go on most of tomorrow and into the week.


- tom_a_sparks - 2010-08-31

I watch a show called All Saints
the subtitle for the episode is "End game"
mythicalLibraian gave it season 02 Episode 25
yet the last episode recorded was from season 10

looking at the list there are 2 episode with "End game"
http://thetvdb.com/?tab=seasonall&id=71815&lid=7

Is there any way to guesstimate the season that Episode is in based on the pasts Episodes?


- outleradam - 2010-09-01

This should have been fixed in last night's SVN revision session. The problem was mythicalLibrarian was matching based on original airdate, then verifying it's fuzzy logic capabilities (just to throw a "fuzzy does not jive with airdate" message in the logs), but it was not resetting the match to the airdate. The problem is fixed and mythicalLibrarian should correctly identify that episode now. Fuzzy should only be used for a backup when original airdate cannot be matched.


- outleradam - 2010-09-01

I just checked and the original airdate is not present for that episode. The original airdate needs to be updated.

The SVN is back up and running. I will commit SVN to stable as soon as I am confident that everything is operating properly.


- outleradam - 2010-09-03

As of:
19 min ago r547 (bypass download support)

mythicalSetup will no longer ask you to build. This was a redundant step and is now elimitated. The first mythicalSetup screen will now ask you if you want to download Latest, Stable, or Build using current and you will then begin the process from there. No more download and then build.

Today many errors were fixed which were mainly just disconcerting and did not apply to the mode which mythicalLibrarian was currently operating. Such as failure to create a folder when in use original dir mode, or echoing of commands which should have been redirected as a comment.

So, if there are no complaints over the weekend, SVN will become stable on Sunday.

Please let me know if there's anything I can do to make the next stable release better. Just let me know if there are any problems.


- outleradam - 2010-09-05

Let me know if there are any problems. Even the smallest change is worth it if it enhances usability.

http://code.google.com/p/mythicallibrarian/updates/list


- outleradam - 2010-09-05

mythicalLibrarian Revision 555 committed to stable.

The SVN will now focus on automating doover jobs. Anything mythicalLibrarian determines a doover job will be scheduled for doover with POSIX at.

The SVN should be considered unstable as scheduling jobs can lead to redundance and memory leaks. Since it's inception, mythicalLibrarian has run to completion and stopped. Starting with the next few commits, mythicalLibrarian will begin scheduling jobs.

mythicalLibrarian will now be considered a fully automated media manager. It will be some time before the next SVN gets committed to stable.

Let me know of any undesired operation or problems you may encounter


- thrasherx - 2010-09-06

I'm getting the "ERROR: User job returned non-zero. Check Logs." message on my mythweb backend status. My output.log tells me:
Code:
###################DEBUG MODE ENABLED####################
MY USER NAME:mythtv-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/mythicalLibrarian-
MOVE DIR:/home/media-server/Videos/TV-Recorded- USING SHOWNAME AS FOLDER:Disabled-
FAILSAFE MODE:Disabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/media-server/Videos/TV-Recorded-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Enabled DEBUG MODE:Enabled-
INPUT SHOW NAME:Family Guy- LOCAL SHOW NAME TRANSLATION:- ShowName:Family Guy
DATABASE UPDATED:- TVDB LAST UPDATED:1283810478- CURRENT:9999999-
RESOLVED SERIES ID:75978- RESOVED SHOW NAME:Family Guy-
INPUT EPISODE NAME:No Chris Left Behind- ABSOLOUTE EPISODE NUMBER:161- RESOLVED EPISODE NAME:No Chris Left Behind-
SEASON:S05- EPISODE:E16- SYMLINK MODE:LINK- FILESIZE: 0kB-
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:5695860kB- WORKINGDIR:1 FREE:5695860kB-
[b]MOVEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVEDIR:0- FREE:kB-[/b]
PRIMARYSHOWDIRWRITABLE:1-  FREE:5695860kB-ALTERNATESHOWDIRWRITABLE:1- FREE:5695860kB-
PRIMARYMOVIEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVIEDIR:0- FREE:5695860kB-
DATABASE TYPE:schedulesdirect1-
RECSTART:2010-09-05 18:30:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2007-05-06-
PROGRAMID:EP002960010098- CHANNEL ID:1009- CATEGORY:Sitcom- GOFORDOOVER:1-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:296001- MATCHED TVDB SERIES ID:75978-
PLOT: The family enrolls Chris in a snooty boarding school; the chicken returns to pick a fight with Peter.
####################END OF DEBUG LOG#####################
CREATING FOLDER: /home/media-server/Videos/TV-Recorded
COULD NOT CREATE /home/media-server/Videos/TV-Recorded/Family Guy
##########################################################
#############FAILSAFE MODE HAS BEEN ACTIVATED#############
##########################################################
FAILSAFE FLAG WAS SET CHECK PERMISSIONS AND FOLDERS

My "MOVEDIR" doesn't appear to be writeable. I've ran sudo chmod a+rw on the output folder, but haven't had any luck. I'm able to run the script just fine through my regular user account, but have verified that the mythtv group can't (sudo su mythtv). I'm a little confused how there's a mythtv group but there isn't a user named mythtv. Any help?


- outleradam - 2010-09-06

It appears that you are running an old version of mythicalLibrarian. The newer ones have the SVN or stable version in the log.
Code:
@@@@@@@@@@@NEW SEARCH INITIATED AT Sun Sep  5 16:18:05 CDT 2010@@@@@@@@@@@@@@
Revision 552 Fri Sep 3 14:46:06 CDT 2010 on Linux


Please run the following.
Code:
sudo mythicalLibrarian --update  #to create the setup dir.
cd ~/.mythicalLibrarian/mythicalSetup
sudo mythicalLibrarian --update
mythicalLibrarian --doMaintenance

It should fix your problems if you read the new updated messages. Usually you would just run mythicalLibrarian --update, but since you are updating from an old version, the --doMaintenance flushes the database and old cache logs for you.

Make sure user mythtv has access to read and write in that folder as well.


- outleradam - 2010-09-06

Also that brings up a good point... The filesystem checks do a great job of telling you that things work or don't, but they don't have descriptive messages as to why. I will increase the error logging for filesystem checks.


- outleradam - 2010-09-06

thrasherx: R557 took the checkpermissions routine from this:
Code:
checkpermissions () {
     TMoveDirWritable=0
     if [ "$2" != "" ] && [ "$1" != "" ] && [ $1 -lt $2 ]; then
         echo "Testing write permission on $3">$3/arbitraryfile.ext
         if [ -f "$3/arbitraryfile.ext" -a -s "$3/arbitraryfile.ext" ]; then
             rm -f "$3/arbitraryfile.ext"
             test ! -f "$3/arbitraryfile.ext" && TMoveDirWritable=1 || TMoveDirWritable=0
         else
             TMoveDirWritable=0
             echo "CHECK PERMISSIONS ON $MoveDir"
         fi
     elif [ -z "$2" ] || [ $1 -ge $2 ]; then
          TMoveDirWritable=0
         echo "UNUSABLE SPACE-CHECK:$3"
    fi
     return 0
}
to this:

Code:
checkpermissions () {
     TMoveDirWritable=0

###DEBUG Increase logging level
     if [ "$2" != "" ]; then
          if [ "$1" != "" ]; then
             if [ -d "$3" ]; then
                 if [ $1 -lt $2 ]; then
                     echo "Testing write permission on $3">$3/arbitraryfile.ext
                     if [ -f "$3/arbitraryfile.ext" ] ; then
                         if [ -s "$3/arbitraryfile.ext" ]; then
                             rm -f "$3/arbitraryfile.ext"
                             test ! -f "$3/arbitraryfile.ext" && TMoveDirWritable=1 || TMoveDirWritable=0
                         else
                             TMoveDirWritable=0
                             echo "CREATED $3/arbitraryfile.ext BUT COULD NOT WRITE DATA INTO THE FILE"
                             echo "CREATED $3/arbitraryfile.ext BUT COULD NOT WRITE DATA INTO THE FILE">"$mythicalLibrarian/output.log"
                        fi
                    else
                          echo "$MyUserName TRIED TO WRITE TO $3 --FAILED"
                        test ! -r "$3" && echo "QUICK FIX: sudo chmod a+r \"$3\""
                         test ! -w "$3" && echo "QUICK FIX: sudo chmod a+w \"$3\""
                            echo "QUICK FIX: sudo chown $MyUserName \"$3\""
                          echo "$MyUserName TRIED TO WRITE TO $3 --FAILED">"$mythicalLibrarian/output.log"
                        test ! -r "$3" && echo "QUICK FIX: sudo chmod a+r \"$3\"">"$mythicalLibrarian/output.log"
                         test ! -w "$3" && echo "QUICK FIX: sudo chmod a+w \"$3\"">"$mythicalLibrarian/output.log"
                            echo "QUICK FIX: sudo chown $MyUserName \"$3\"">"$mythicalLibrarian/output.log"
                     fi
                 elif [ -z "$2" ] || [ $1 -ge $2 ]; then
                      TMoveDirWritable=0
                     echo "INSUFFICIENT FREE SPACE ON $3"
                     echo "QUICK FIX: Remove unnecessary files"
                     echo "INSUFFICIENT FREE SPACE ON $3">"$mythicalLibrarian/output.log"
                     echo "QUICK FIX: Remove unnecessary files">"$mythicalLibrarian/output.log"
                fi
            else
                echo "FOLDER DOES NOT EXIST: $3"
                echo "QUICK FIX: mkdir \"$3\""
                echo "FOLDER DOES NOT EXIST: $3">"$mythicalLibrarian/output.log"
                echo "QUICK FIX: mkdir \"$3\"">"$mythicalLibrarian/output.log"
             fi

        else
            echo "COULD NOT READ FILESIZE ON $InputPath"
            echo "QUICK FIX: chmod -R 666 \"$InputPath\""
            echo "COULD NOT READ FILESIZE ON $InputPath">"$mythicalLibrarian/output.log"
            echo "QUICK FIX: chmod -R 666 \"$InputPath\"">"$mythicalLibrarian/output.log"
         fi
    else
         echo "COULD NOT READ FREE SPACE on \"$originaldirname\""
        echo "QUICK FIX: chmod -R 666 \"$originaldirname\""
        echo "COULD NOT READ FREE SPACE on \"$originaldirname\"">"$mythicalLibrarian/output.log"
        echo "QUICK FIX: chmod -R 666 \"$originaldirname\"">"$mythicalLibrarian/output.log"
    fi

     return 0
}
It is my hope that these changes will tell you precisely the commands to run from the terminal to make everything work properly. Let me know if there are any problems.


- thrasherx - 2010-09-06

Unfortunately, still nothing Sad When I run "--mythicalDiagnostics" it creates the Mr. Rogers folder just fine, but I'm assuming because i'm not doing it from the mythtv user. Here's my new output.log. I've tried the chmod -R 666 as well as chmod -R a+rw, but it still doesn't work.

Code:
QUICK FIX: chmod -R 666 "/var/lib/mythtv/recordings"
###################DEBUG MODE ENABLED####################
MY USER NAME:mythtv-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/.mythicalLibrarian-
MOVE DIR:/home/media-server/Videos/TV-Recorded- USING SHOWNAME AS FOLDER:Enabled-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/media-server/Videos/TV-Recorded-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Enabled
INPUT SHOW NAME:The National Parks: America's Best Idea- LOCAL SHOW NAME TRANSLATION:- ShowName:The National Parks: America's Best Idea
DATABASE UPDATED:- TVDB LAST UPDATED:1283856183- CURRENT:1283799558-
RESOLVED SERIES ID:116171- RESOVED SHOW NAME:The National Parks: America's Best Idea-
INPUT EPISODE NAME:The Empire of Grandeur (1915-1919)- ABSOLOUTE EPISODE NUMBER:9- RESOLVED EPISODE NAME:The Empire of Grandeur (1915-1919)-
SEASON:S01- EPISODE:E03- SYMLINK MODE:MOVE- FILESIZE: 0kB-
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:9273908kB- WORKINGDIR:1 FREE:9273908kB-
MOVEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVEDIR:0- FREE:kB-
PRIMARYSHOWDIRWRITABLE:0-  FREE:kB-ALTERNATESHOWDIRWRITABLE:0- FREE:kB-
PRIMARYMOVIEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVIEDIR:0- FREE:9273908kB-
DATABASE TYPE:schedulesdirect1-
RECSTART:2010-09-06 12:50:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2009-09-29-
PROGRAMID:EP011701200003- CHANNEL ID:1007- CATEGORY:Documentary- GOFORDOOVER:1-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:1170120- MATCHED TVDB SERIES ID:116171-
PLOT: The National Park Service is established in 1916; Stephen Mather campaigns to establish the Grand Canyon in Arizona as a national park.
####################END OF DEBUG LOG#####################
CREATING FOLDER: /home/media-server/Videos/TV-Recorded/The National Parks Americas Best Idea
COULD NOT CREATE /home/media-server/Videos/TV-Recorded/The National Parks Americas Best Idea/Season 1/The National Parks Americas Best Idea
##########################################################
#############FAILSAFE MODE HAS BEEN ACTIVATED#############
##########################################################



- outleradam - 2010-09-06

Those error messages are important. I can see that permissions are not set correctly. You need to run the commands given in the program. You should have full access at this point. Did you run the commands with sudo to elevate permissions?

Code:
sudo chmod -R 666 "/var/lib/mythtv/recordings"

mythtv does not have permission to run the following:
Code:
mkdir /home/media-server/Videos/TV-Recorded/The National Parks Americas Best Idea

per chance do you run as root on your system when creating folders? This would make the user and group of the paths you're using "root" which would make mythtv not able to access them.

Is /home/media-server a network attached storage drive? If so, then post your /etc/fstab so I can see if the permissions are set correctly for mythtv's use.

here is my NAS entry, eg:
Code:
//192.168.1.1/public /home/mythtv/NAS cifs credentials=/home/adam/.smbcredentials,iocharset=utf8,dir_mode=0777,file_mode=0777 0 0

I need to see the entire section of the log from start to finish. starting with "New Search" and ending right before the end of the log or "New Search" You're cropping important information out.

it's always best to use the "use original dir" option, then share the dir over the network. this way mythtv owns the folder and the operating system shares the folder.

This is some sort of multiuser configuration issue.
So here's what I need:
1. results of ls -l /var/lib/mythtv/recordings
2. results of ls -l /home/media-server/Videos/TV-Recorded/
3. results of cat /etc/fstab
4. the last log entry result of cat ~/home/mythtv/.mythicalLibrarian/output.log


- thrasherx - 2010-09-06

outleradam Wrote:Those error messages are important. I can see that permissions are not set correctly. You need to run the commands given in the program. You should have full access at this point. Did you run the commands with sudo to elevate permissions?

Code:
sudo chmod -R 666 "/var/lib/mythtv/recordings"
I've run this without success.

outleradam Wrote:per chance do you run as root on your system when creating folders? This would make the user and group of the paths you're using "root" which would make mythtv not able to access them.
Nope, I created it using my main user account.

outleradam Wrote:Is /home/media-server a network attached storage drive? If so, then post your /etc/fstab so I can see if the permissions are set correctly for mythtv's use.
No, it's a mounted Raid 5 array.

outleradam Wrote:I need to see the entire section of the log from start to finish. starting with "New Search" and ending right before the end of the log or "New Search" You're cropping important information out.
That was the entire entry. I was perplexed as to why the "New Search" phrase was missing, but it's your latest SVN from today (559).

outleradam Wrote:1. results of ls -l /var/lib/mythtv/recordings
-r-xr-xr-x 1 mythtv mythtv 4808431068 2010-09-06 00:35 1002_20100905233500.mpg
-r-xr-xr-x 1 mythtv mythtv 2815126664 2010-09-05 19:00 1006_20100905183000.mpg
-r-xr-xr-x 1 mythtv mythtv 2099563132 2010-09-05 20:00 1006_20100905193700.mpg
dr-xr-sr-x 2 mythtv mythtv 4096 2010-09-06 12:30 Episodes
dr-xr-sr-x 2 mythtv mythtv 4096 2010-09-06 12:30 Movies
dr-xr-sr-x 2 mythtv mythtv 4096 2010-09-06 12:30 Showings
dr-xr-sr-x 2 root mythtv 4096 2010-09-05 23:34 working

outleradam Wrote:2. results of ls -l /home/media-server/Videos/TV-Recorded/
drwxrwxrwx 2 mythtv robot 4096 2010-09-06 14:36 TV-Recorded

outleradam Wrote:3. results of cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=a7a755ac-e854-477a-b58c-55f0cb3ce229 / ext4 errors=remount-ro 0 1
# /home/media-server was on /dev/md1 during installation
UUID=b0980baa-6365-4f65-80f3-03b7f502c95b /home/media-server ext4 defaults 0 2
# swap was on /dev/sda5 during installation
UUID=089a71bc-b7e0-4a89-8ef1-e8e5f48c61f6 none swap sw 0 0

outleradam Wrote:4. the last log entry result of cat ~/home/mythtv/.mythicalLibrarian/output.log
Code:
QUICK FIX: chmod -R 666 "/var/lib/mythtv/recordings"
###################DEBUG MODE ENABLED####################
MY USER NAME:mythtv-
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/.mythicalLibrarian-
MOVE DIR:/home/media-server/Videos/TV-Recorded- USING SHOWNAME AS FOLDER:Enabled-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/media-server/Videos/TV-Recorded-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Enabled
INPUT SHOW NAME:The National Parks: America's Best Idea- LOCAL SHOW NAME TRANSLATION:- ShowName:The National Parks: America's Best Idea
DATABASE UPDATED:- TVDB LAST UPDATED:1283856183- CURRENT:1283799558-
RESOLVED SERIES ID:116171- RESOVED SHOW NAME:The National Parks: America's Best Idea-
INPUT EPISODE NAME:The Empire of Grandeur (1915-1919)- ABSOLOUTE EPISODE NUMBER:9- RESOLVED EPISODE NAME:The Empire of Grandeur (1915-1919)-
SEASON:S01- EPISODE:E03- SYMLINK MODE:MOVE- FILESIZE: 0kB-
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:9273908kB- WORKINGDIR:1 FREE:9273908kB-
MOVEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVEDIR:0- FREE:kB-
PRIMARYSHOWDIRWRITABLE:0-  FREE:kB-ALTERNATESHOWDIRWRITABLE:0- FREE:kB-
PRIMARYMOVIEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVIEDIR:0- FREE:9273908kB-
DATABASE TYPE:schedulesdirect1-
RECSTART:2010-09-06 12:50:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2009-09-29-
PROGRAMID:EP011701200003- CHANNEL ID:1007- CATEGORY:Documentary- GOFORDOOVER:1-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:1170120- MATCHED TVDB SERIES ID:116171-
PLOT: The National Park Service is established in 1916; Stephen Mather campaigns to establish the Grand Canyon in Arizona as a national park.
####################END OF DEBUG LOG#####################
CREATING FOLDER: /home/media-server/Videos/TV-Recorded/The National Parks Americas Best Idea
COULD NOT CREATE /home/media-server/Videos/TV-Recorded/The National Parks Americas Best Idea/Season 1/The National Parks Americas Best Idea
##########################################################
#############FAILSAFE MODE HAS BEEN ACTIVATED#############
##########################################################
FAILSAFE FLAG WAS SET CHECK PERMISSIONS AND FOLDERS
PERMISSION ERROR OR DRIVE FULL
ATTEMPTING SYMLINK TO FAILSAFE DIR: /home/mythtv/FailSafe
FAILSAFE MODE COMPLETE: SYMLINK CREATED
FAILSAFE MODE FAILURE CHECK PERMISSIONS AND FREE SPACE IN /home/mythtv/FailSafe
@@@@@@@@@@@NEW SEARCH INITIATED AT Mon Sep  6 14:50:48 EDT 2010@@@@@@@@@@@@@@
Revision 559 Mon Sep 6 12:47:29 EDT 2010 on Linux