Kodi Community Forum
mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings (/showthread.php?tid=65644)



- outleradam - 2010-02-02

I just don't know fidora. If you are using the same dependencies then it should work the same. What version of bash again? Something is screwy here. All the inputs to checkpermissions are working but the outputs are comming back wrong. Does fidora have online test command? I can't get a ver on that. It's like it's testing good and then setting the 0 flag. I've got to go to bed. If you can, try testing that test command and see where it goes broke.


- watson540 - 2010-02-02

i upgraded bash the other day when i was messing with that wrong agrep version..im running gentoo not fedora..and an online test command i dunno what you mean but test is built into bash afaik
I have read bash scripting tutorials before so i know a little about the brackets [] meaning test and all that..all i can do is mess around and upgrade/downgrae stuff i guess
hope the debug log on pastebin helps
my bash version and use flags currently
[i] app-shells/bash
Available versions: 3.1_p17 3.2_p39 ~3.2_p48-r1 4.0_p35 **4.1 {afs bashlogger examples mem-scramble net nls plugins vanilla}
Installed versions: 4.0_p35(04:10:04 PM 01/28/2010)(net nls -afs -bashlogger -examples -mem-scramble -plugins -vanilla)
Homepage: http://tiswww.case.edu/php/chet/bash/bashtop.html
Description: The standard GNU Bourne again shell


- watson540 - 2010-02-02

hmm this must convince you..you introduced a bug (for me anyway) in your latest version, as I have rolled back to an earlier version i had on my drive and lo and behold ....
Code:
myth mythicalLibrarian # ./mythicalLibrarian.sh "Family Guy" "Da Boom" "/mnt/ntfs/myth/1015_20100201220000.mpg"
./mythicalLibrarian.sh: line 526: /home/mythtv/mythicalLibrarian/dir.tracking: No such file or directory
mv: cannot stat `/home/mythtv/mythicalLibrarian/dir.tracking2': No such file or directory
Accessing MythTV DataBase:
COMMERCIAL DATA START:9649 STOP:14341
COMMERCIAL DATA START:24246 STOP:30292
COMMERCIAL DATA START:42080 STOP:46625
0
RECSTART:2010-02-01 22:00:00 MOVIEYEAR:1999 SERIESDATE:1999-12-26
PROGRAMID:EP002960010012 ShowCategory:Comedy STARS:0 RATING:0
PLOT: Paranoid about Y2K, Peter herds the family into the basement for protection.
TESTING FOR ZAP2IT SERIES ID MATCH:Family Guy
DATABASE IS MAINTAINED. TIME IS:1265094396 NEXT UPDATE IS:1265160381
No direct Zap2it match found. Please update TheTvDb.com
USING FUZZY LOGIC FOR EPISODE RECOGNITION
Family Guy
FUZZY LOGIC SHOW NAME: Family Guy ID: 75978
./mythicalLibrarian.sh: line 425: [: : integer expression expected
./mythicalLibrarian.sh: line 467: [: : integer expression expected
FUZZY Exx NUMBER:35
EPISODE:Da Boom ABSOLUTE NUMBER:35 S02E03
FILE SIZE:584738kB
MOVEDIR FREE SPACE: 89131720kB- WRITABLE:1
ALTMOVEDIR FREE SPACE: 28225008kB- WRITABLE:1
MOVIEDIR FREE SPACE: 28225008kB- WRITABLE:1
ALTMOVIEDIR FREE SPACE: 89131656kB- WRITABLE:1
PLOT: Paranoid about Y2K, Peter herds the family into the basement for protection.
SET TARGET DIR AS: /mnt/ntfs/tv
CREATING LINK IN LINK MODE
myth mythicalLibrarian # ls -la /mnt/ntfs/tv
total 108
drwxrwxrwx 1 root root   4096 2010-02-02 02:01 .
drwxrwxrwx 1 root root 106496 2010-02-01 20:38 ..
drwxrwxrwx 1 root root      0 2010-02-02 02:01 Family Guy
myth mythicalLibrarian # ls -la /mnt/ntfs/tv/Family\ Guy/
total 5
drwxrwxrwx 1 root root    0 2010-02-02 02:01 .
drwxrwxrwx 1 root root 4096 2010-02-02 02:01 ..
lrwxrwxrwx 1 root root   84 2010-02-02 02:01 Family Guy.S02E03 (Da Boom).mpg -> /mnt/ntfs/myth/1015_20100201220000.mpg
-rwxrwxrwx 1 root root   85 2010-02-02 02:01 Family Guy.S02E03 (Da Boom).txt
myth mythicalLibrarian #
granted..the "integer expected error" is still there but as you see..no failsafe activated and it made the files it needed to..the integer expected i imagine may be cause i use mc2xml..i hope

pay no mind to the owner..i ran it as root the first time but im gonna change those permisssions..and they werte fine before I ran it as well


- watson540 - 2010-02-02

do you even use edl? your edl files dont work nothing comes up on screen efter edl: on the osd
however i read that mtyh:// functionality auto pulls the edl info from the backend.so i tried it..yeah ..edl info on the osd, but it doesnt skip

however.open it using dvdplayer instead of mplayer and it DOES skip commercials. this is under Mythtv though not using the "library" cause thats what your app does and the edl files must be wrong unless someone says its working for them..
i can open the files in the library with either player (the files your script adds to the library) and no edl info at all on the osd


- outleradam - 2010-02-02

if you would, please e-mail a copy of that file to me. my email is at the top of mythicalLibrarian.sh.


- watson540 - 2010-02-02

done


- outleradam - 2010-02-03

The only thing I really changed in that release is the rm command remove the -F and see if that changes anything.

I'm at work right now, do you know of a place I can get a live ISO of your OS to work with and test out mythicalLibrarian? I would like to see the problem for myself and see if I can work it from your side.


- outleradam - 2010-02-03

I just realized that you were running in LINK mode. Run in MOVE mode and it will be fully compatable with NTFS. The symlinks let mythtv keep it's control over the files, and they are kept on the mythtv drive. FAT32 and i'm pretty sure NTFS do not support symlinks AFAIK.

MOVE cleans up after itself. LINK mode does not.


- outleradam - 2010-02-03

watson540 Wrote:http://pastebin.org/85299
Could I get you to post the rest of the run? It cut off after it ran the first little bit, It didn't even get to the processing section, it set variables and started defining the functions then just ended.

This seems like a much better way of doing it then for me to run fidora and muck around with it.

watson540 Wrote:pay no mind to the owner..i ran it as root the first time but im gonna change those permisssions..and they werte fine before I ran it as well
Wait, you mean that all the working files were created by root the first time? If the working files are created by root, then you may not be able to overwrite them.


there is nothing in the diff between the current version and the one you have that would cause that problem.

Just to be completely sure, try this SVN link. http://mythicallibrarian.googlecode.com/svn/trunk/


- watson540 - 2010-02-03

dont know of any recent "live" build of gentooo.[EDIT] duh FOR ME, TRY SABAYON if you want a live gentoo build..itsbeautiful ..I used it bfore I switched fully to gentoo .i know a few of the gentoo cd's can run live though, they have X and run xfwm i think

i tried the script in move mode. iwas baffled when it actualy made the symlinks on my ntfs drive..bad part about move mode was that I was resetting your scriipt all the time and it was ripping mythbackends videos from its clutches..s o if your script screws up..i end up DELETIUNG a whole day or more recordings to start over

but be assured ive tried move mode on both ext3 and ntfs...and like i said earlier pay no mind to the owner root I only did that once to see if it works, then I erased everyhing from working folders and started over

I wish someone else would come on here and help out..ill get you some more info soon


- watson540 - 2010-02-03

later on ill try to run both in bash debug mode and ill email you the results..pastebin doesnt like big pastes


- am4c130d - 2010-02-03

I appear to have had the same problem as Watson540, but from this thread tracked down where the script was failing for me.

I previously had an earlier version of MythicalLibrarian that simply queried TheTVDB and if the show didn't have any episode information it failed - but where the info was present and on TheTVDB it worked great. I updated today to a version dated 1/29/10, and immediately ran in to this problem.

I was trying to run the script against two shows which did not have episode names, or plot descriptions from SchedulesDirect, and the MOVE had failed last night with the older script. With the new script, like the examples OutlerAdam gave back on #121 I had adequate disk space, but the directory showed as not writable.

Code:
FILE SIZE:485964kB
MOVEDIR FREE SPACE: 203782276kB- WRITABLE:0
ALTMOVEDIR FREE SPACE: 203782276kB- WRITABLE:0
MOVIEDIR FREE SPACE: 203782276kB- WRITABLE:0
ALTMOVIEDIR FREE SPACE: 203782276kB- WRITABLE:0

I played with the config parameters to no avail, and then started looking at the script itself (and I am no script writer - certainly I could not dream of writing anything as complex as this).

Looking at the checkpermissions subroutine I inverted the values of TMoveDirWritable in the second "if" test - which should force a write, where previously it indicated the space was not writable. However, when I ran the script I received the same WRITABLE: 0. This indicated that the subroutine was not been executed - though it appeared that it was being called. Next I forced the initialization of TMoveDirWritable to 1, and this gave this result

Code:
FILE SIZE:485964kB
MOVEDIR FREE SPACE: 203782276kB- WRITABLE:1
ALTMOVEDIR FREE SPACE: 203782276kB- WRITABLE:1
MOVIEDIR FREE SPACE: 203782276kB- WRITABLE:1
ALTMOVIEDIR FREE SPACE: 203782276kB- WRITABLE:1

Which again indicated the second "if" test was not actually being run - as the test conditions made no difference to the initialization value.

The first "if" statement, whilst being understandable, made no sense to me - if I read it correctly it says the following

if "episode name" is NULL AND "program name" is NOT NULL AND "program name" is less than "episode name" then run the tests.

Suffice it to say, I commented out that line, and the associated elif and fi etc. as below

Code:
#####CHECK PERMISSIONS#####
#CheckPermissions by writing a small file then deleting it, checking along the way.
#CheckPermissions takes file size, free space on dir, and the dir, then it performs
#tests.  the result will be $TMoveDirWritable as a 1 or a 0 for writable or not.
checkpermissions () {
TMoveDirWritable=0
#  if [ "$2" = "" ] && [ "$1" != "" ] && [ $1 -lt $2 ]; then
        echo "Testing write permission on $3">$3/arbitraryfile.ext
        if [ -f "$3/arbitraryfile.ext" -a -s "$3/arbitraryfile.ext" ]; then
                rm -f "$3/arbitraryfile.ext"
                test ! -f "$3/arbitraryfile.ext" && TMoveDirWritable=1 || TMoveDirWritable=0
        else
                TMoveDirWritable=0
                echo "CHECK PERMISSIONS ON $MoveDir"
        fi
# elif [ -z "$2" ] || [ $1 -ge $2 ]; then
#       TMoveDirWritable=0
#      echo "UNUSABLE SPACE-CHECK:$3"
#  fi
}

And everything now works like a champ. I suspect the correct fix is to replace -lt with -gt, but it still seems meaningless as a non-null program name will always be greater than a null episode name - no?

Out of interest, why would this test be included here - it seems irrelevant, as the program name and episode name seem to have no bearing on whether the disk space is adequate and writable?

However, I hope that this helps - it caused me some head scratching, and the efforts you two have exerted indirectly pointed me in the right direction.

OutlerAdam - thanks for this script, I have a few more old programs to check, but you may just have weened me off Myth frontend after 7 years of happiness and tears.

Thanks

Alan


- outleradam - 2010-02-03

In a function, the variables which are passed become $1, $2, $3......

The variables passed are $1=needed free space, $2 = actual free space, $3 = dirname to be checked.
Code:
checkpermissions "$MoveFileSize" "$MoveDirFreeSpace" "$MoveDir"
When it runs through the checkpermissions function, it checks free space and writability.

This function is designed to be helpful, but I guess it is hurting.

I assume that you are running fidora as well?

Tomorrow I will rewrite that section so as to assign proprietary names to those variables before calling the function instead of calling the passed name.

This is the only Episode which has failed on me in the last 200 runs of mythicalLibrarian. This Episode failed because it is actually teenage mutant ninja turtles 2003 and it was detected as 1987. I added the proper zap2it id for this show in the tvdb.com and I expect the next run will be sucessful.
Code:
@@@@@@@@@@@NEW SEARCH INITIATED AT Sat Jan 30 10:30:49 CST 2010@@@@@@@@@@@@@
%%%NO DATABASE INFORMATION. CHECK LOGIN/PASS OR FILE %%%%%
SEARCHING: www.TheTvDb.com SHOW NAME: Teenage Mutant Ninja Turtles EPISODE: More Worlds Than One
FILE NAME: /home/mythtv/Videos/6012_20100130100000.mpg
SEARCH FOUND:Teenage Mutant Ninja Turtles (1987) ID#: 74582
LOCAL DATABASE UPDATED:/home/mythtv/mythicalLibrarian/Teenage Mutant Ninja Turtles (1987)
SEARCH FOUND:Teenage Mutant Ninja Turtles (2003) ID#: 71032
USING FUZZY LOGIC FOR EPISODE RECOGNITION
No direct Zap2it match found. Please update TheTvDb.com
SEARCH FOUND:Teenage Mutant Ninja Turtles (1987) ID#: 74582
DEFINED ABSOLOUTE EPISODE NUMBER:
###################DEBUG MODE ENABLED####################
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/mythicalLibrarian-
MOVE DIR:/home/mythtv/NAS/Video/Episodes- USING SHOWNAME AS FOLDER:Enabled-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/mythtv/Episodes-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Enabled DEBUG MODE:Enabled-
INPUT SHOW NAME:Teenage Mutant Ninja Turtles- LOCAL SHOW NAME TRANSLATION:Inactive- ShowName:Teenage Mutant Ninja Turtles
SENT TVDB SHOW NAME:Teenage%20Mutant%20Ninja%20Turtles-
RESOLVED SERIES ID:74582- RESOVED SHOW NAME:Teenage Mutant Ninja Turtles (1987)-
INPUT EPISODE NAME:More Worlds Than One- ABSOLOUTE EPISODE NUMBER:- RESOLVED EPISODE NAME:More Worlds Than One-
SEASON:- EPISODE:- SYMLINK MODE:MOVE- FILESIZE: 840784'kB'
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:34623108kB- WORKINGDIR:1 Free:34623108kB-
MOVEDIR:0- FREE:kB- ALTERNATEMOVEDIR:1- FREE:34623108kB-
PRIMARYMOVIEDIRWRITABLE:0- FREE:kB- ALTERNATEMOVIEDIR:1- FREE:34623108kB-
DATABASE INFORMATION
RECSTART:2010-01-30 10:00:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2008-03-08-
PROGRAMID:EP005279840124- CHANNEL ID:6012- CATEGORY:Children-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:EP00527984- MATCHED TVDB SERIES ID:-
####################END OF DEBUG LOG#####################
%%%%%%%%%%www.TheTvDB.com information is incomplete Teenage Mutant Ninja Turtles, More Worlds Than One
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%Please consider helping out and adding to thetvdb%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As you can see in this log, my NAS was down, so mythicalLibrarian marked the NAS movedir and primarymoviedir, as non-writable. That's my only error in my log.
They usually come out like this
Code:
@@@@@@@@@@@NEW SEARCH INITIATED AT Tue Feb  2 20:01:04 CST 2010@@@@@@@@@@@@@
%%%NO DATABASE INFORMATION. CHECK LOGIN/PASS OR FILE %%%%%
SEARCHING: www.TheTvDb.com SHOW NAME: Scrubs EPISODE: My Finale
FILE NAME: /home/mythtv/NAS/Video/6056_20100202193000.mpg
SEARCH FOUND:Scrubs ID#: 76156
USING FUZZY LOGIC FOR EPISODE RECOGNITION
No direct Zap2it match found. Please update TheTvDb.com
SEARCH FOUND:Scrubs ID#: 76156
DEFINED ABSOLOUTE EPISODE NUMBER: 214
###################DEBUG MODE ENABLED####################
LISTING INTERNAL VARIABLES USED BY mythicalLibrarian.
INTERNET TIMEOUT:50- TVDB API KEY:6DF511BB2A64E0E9- mythicalLibrarian WORKING DIR:/home/mythtv/mythicalLibrarian-
MOVE DIR:/home/mythtv/NAS/Video/Episodes- USING SHOWNAME AS FOLDER:Enabled-
FAILSAFE MODE:Enabled- FAILSAFE DIR:/home/mythtv/FailSafe- ALTERNATE MOVE DIR:/home/mythtv/Episodes-
USE ORIGINAL DIR:Disabled NOTIFICATIONS:Enabled DEBUG MODE:Enabled-
INPUT SHOW NAME:Scrubs- LOCAL SHOW NAME TRANSLATION:Inactive- ShowName:Scrubs
SENT TVDB SHOW NAME:Scrubs-
RESOLVED SERIES ID:76156- RESOVED SHOW NAME:Scrubs-
INPUT EPISODE NAME:My Finale- ABSOLOUTE EPISODE NUMBER:214- RESOLVED EPISODE NAME:My Finale-
SEASON:S08- EPISODE:E18- SYMLINK MODE:MOVE- FILESIZE: 515609'kB'
CREATE AND DELETE FLAGS: ORIGINALDIR:1- FREE:684075584kB- WORKINGDIR:1 Free:33352652kB-
MOVEDIR:1- FREE:684075584kB- ALTERNATEMOVEDIR:1- FREE:33352652kB-
PRIMARYMOVIEDIRWRITABLE:1- FREE:684075584kB- ALTERNATEMOVIEDIR:1- FREE:33352652kB-
DATABASE INFORMATION
RECSTART:2010-02-02 19:30:00- MOVIE YEAR:- ORIGINAL SERIES DATE:2009-05-06-
PROGRAMID:EP004461600173- CHANNEL ID:6056- CATEGORY:Sitcom-
EXTRAPOLATED DATA DETERMINED THIS RECORDING AS A:Series With Episode Data- STARS:0 RATING:0
ZAP2IT SERIES ID:EP00446160- MATCHED TVDB SERIES ID:-
####################END OF DEBUG LOG#####################
MOVING FILE: /home/mythtv/NAS/Video/6056_20100202193000.mpg to /home/mythtv/NAS/Video/Episodes/Scrubs/Scrubs.S08E18 (My Finale).mpg
@@@@@@@@@@@@@OPERATION COMPLETE Tue Feb 2 20:01:12 CST 2010 @@@@@@@@@@@@@@@@


Ok, so I think i've got it figured out and I will make this change as a work around for certain OS's handling of variable passing.


- watson540 - 2010-02-03

am4c130d Wrote:i appear to have had the same problem as watson540, but from this thread tracked down where the script was failing for me.

I previously had an earlier version of mythicallibrarian that simply queried thetvdb and if the show didn't have any episode information it failed - but where the info was present and on thetvdb it worked great. I updated today to a version dated 1/29/10, and immediately ran in to this problem.

I was trying to run the script against two shows which did not have episode names, or plot descriptions from schedulesdirect, and the move had failed last night with the older script. With the new script, like the examples outleradam gave back on #121 i had adequate disk space, but the directory showed as not writable.

Code:
file size:485964kb
movedir free space: 203782276kb- writable:0
altmovedir free space: 203782276kb- writable:0
moviedir free space: 203782276kb- writable:0
altmoviedir free space: 203782276kb- writable:0

i played with the config parameters to no avail, and then started looking at the script itself (and i am no script writer - certainly i could not dream of writing anything as complex as this).

Looking at the checkpermissions subroutine i inverted the values of tmovedirwritable in the second "if" test - which should force a write, where previously it indicated the space was not writable. However, when i ran the script i received the same writable: 0. This indicated that the subroutine was not been executed - though it appeared that it was being called. Next i forced the initialization of tmovedirwritable to 1, and this gave this result

Code:
file size:485964kb
movedir free space: 203782276kb- writable:1
altmovedir free space: 203782276kb- writable:1
moviedir free space: 203782276kb- writable:1
altmoviedir free space: 203782276kb- writable:1

which again indicated the second "if" test was not actually being run - as the test conditions made no difference to the initialization value.

The first "if" statement, whilst being understandable, made no sense to me - if i read it correctly it says the following

if "episode name" is null and "program name" is not null and "program name" is less than "episode name" then run the tests.

Suffice it to say, i commented out that line, and the associated elif and fi etc. As below

Code:
#####check permissions#####
#checkpermissions by writing a small file then deleting it, checking along the way.
#checkpermissions takes file size, free space on dir, and the dir, then it performs
#tests.  The result will be $tmovedirwritable as a 1 or a 0 for writable or not.
Checkpermissions () {
tmovedirwritable=0
#  if [ "$2" = "" ] && [ "$1" != "" ] && [ $1 -lt $2 ]; then
        echo "testing write permission on $3">$3/arbitraryfile.ext
        if [ -f "$3/arbitraryfile.ext" -a -s "$3/arbitraryfile.ext" ]; then
                rm -f "$3/arbitraryfile.ext"
                test ! -f "$3/arbitraryfile.ext" && tmovedirwritable=1 || tmovedirwritable=0
        else
                tmovedirwritable=0
                echo "check permissions on $movedir"
        fi
# elif [ -z "$2" ] || [ $1 -ge $2 ]; then
#       tmovedirwritable=0
#      echo "unusable space-check:$3"
#  fi
}

and everything now works like a champ. I suspect the correct fix is to replace -lt with -gt, but it still seems meaningless as a non-null program name will always be greater than a null episode name - no?

Out of interest, why would this test be included here - it seems irrelevant, as the program name and episode name seem to have no bearing on whether the disk space is adequate and writable?

However, i hope that this helps - it caused me some head scratching, and the efforts you two have exerted indirectly pointed me in the right direction.

Outleradam - thanks for this script, i have a few more old programs to check, but you may just have weened me off myth frontend after 7 years of happiness and tears.

Thanks

alan

thank you for posting that...now i can fix this bug in the newer script. Thank you for your great insight and wisdom please keep in touch on this thread

aND outleradam thanks for saying youll integrate this change.. i know those of us who use a real linux os (gentoo) Tongue really appreciate it. and sorry for the troubles i give you, just want a working setup thats all

ps..wtf is fidora?? you mean fedora? i havent seen anyone in this thread mention it


- outleradam - 2010-02-03

Hey, instead of just setting tmovdirwritable=1, try removing the $ and the quotes in the variables passed

In the section that looks like this:
Code:
checkpermissions "$MoveFileSize" "$MoveDirFreeSpace" "$MoveDir"

change them to this:
Code:
checkpermissions MoveFileSize MoveDirFreeSpace MoveDir

If that does not work, try removing the quotes and keeping the $. I would like to keep it clean like this, but I can remove the variable passing if required.
Code:
checkpermissions $MoveFileSize $MoveDirFreeSpace $MoveDir
or this
Code:
checkpermissions "$(MoveFileSize)" "($MoveDirFreeSpace)" "$(MoveDir)"

There has to be a way to pass variables in Gentoo. Please test this out and let me know if it sets the flags properly. Just run the program a few times with these variants and see if it properly sets permissions. I don't want to add 20 lines to define variables which should be passed properly if I don't have to. Let me know what works please?


Basically, by setting tmovedirwritable=1 manually, you are bypassing several safety mechanisims which are designed to prevent improper naming, random errors, and data loss.