Kodi Community Forum
mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Supplementary Tools for Kodi (https://forum.kodi.tv/forumdisplay.php?fid=116)
+--- Thread: mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings (/showthread.php?tid=65644)



- outleradam - 2010-12-29

It looks like local policy is preventing you from modifying the file.

In order to preserve MythTV's ownership of the file, mythicalLibrarian's logging and overall security of the system, the best way is to have user mythtv control all aspects of mythicalLibrarian's operation. For this reason, mythicalSetup asks you for a password for user mythtv. This makes user mythtv a full-fledged valid user on your system.

When running as user rmikulec, file operations and logging are performed by that username rather then mythtv. The linux filesystem's granular control can be a burden sometimes. It's simpler to simply not deal with that and allow mythtv to control it's files by running mythicalLibrarian as mythtv. User jobs are automatically run with mythtv permissions. Under most distributions, MythTV is run with mythtv permssions, however some distributions use other usernames, so I could not make it manditory for the user mythtv to be used. Because of Linux's granular permissions, and local policy preventing certain actions on different distributions, multiple users creating multiple files with different permissions is impossible to manage. The way around this is to bypass all of linux security, run as root or allow sudo with no password... However this is bad procedure and allows for a gigantic security hole.

It's always best if the user who creates the file, manages the file. In this case it's mythtv.

You can run
Code:
mythicalLibrarian --undo
sudo su mythtv
mythicalLibrarian --scan mpg /path_to_my/recordings

Let me know if there are any other problems.


- rmikulec - 2010-12-29

Ok, I think I have it figured out. Everything works fine until it attempts to transcode. I think the issue is that there are parentheses "(" in the naming convention and it bombs the transcoding. I havent figured a way around it yet.


- outleradam - 2010-12-29

Put double quotes around the ffmpeg job filenames. "

() is a perfectly valid filenames, the spaces are not. It needs to have "quotes" wrapped around it.

Please email a copy of your userjob, or update the wiki when you are finished. That way this will be easier for others.


- GregoryK - 2010-12-29

Adam:

Is there any way to get mythicalLibrarian to generate commercial-skip files that XBMC Dharma will properly execute? I have followed the related threads regarding the frame-rate issue in Dharma, and it doesn't look like a solution (either via MythTV or XBMC patches) is very probable in the near term.

I've been trying to think of workarounds, so that my wife & kids can use XBMC for playback without running into bad skipping.

Would transcoding the files as a user job in Myth, removing the commercials, before running mythicalLibrarian, be a work-around that makes sense?

If yes, would transcoding also help with my existing recordings, which have already been renamed and moved via mythicalLibrarian? (I suppose I might have to delete the commskip txt files from the /Episodes/[name] directories.)

Any thoughts are appreciated.

/Greg


- outleradam - 2010-12-30

GregoryK: The problem is XBMC's handling of MPEG2-TS variable framerate. The show comes in at 24fps, and some commercials are appended into the broadcast at 60fps. Using MythTranscode on each file will patch the problem in both MythTV and XBMC by setting it at a fixed framerate. MythTV handles variable framerate with a table which is generated by it's comskip handler. If the file is variable framerate, the frame numbers in the comskip.txt are correct, but XBMC blindly assumes fixed framerate based on the first few frames of the recording.

The patch is to have MythTV transcode every file. This creates a fixed framerate for every show. This should be done before mythicalLibrarian runs. So, set MythTV to run only 1 job at a time. Set MythTV to do MythTranscode job, select MythCommFlag checkbox, and have mythicalLibrarian run last.

Let me know if there are any problems


- GregoryK - 2010-12-30

Adam:

Thanks for the reply, good to hear that there is a workaround.

Selecting the MythCommFlag checkbox will have the transcode honor the cutlist automatically and remove the commercials? Or will it just flatten the framerate in the transcoded file (not sure I care either way, just curious.)

Any ideas for applying this workaround (or a similar one) for my existing recordings that have already been processed by mythicalLibrarian? Something like the script described on this page, only run as a cron job (instead of a user job in frontend), maybe?

/Greg


- outleradam - 2010-12-30

cronjobs are used when you want to have unattended processes run at predetermined times. You can attend this process. Off-hand I can't really remember the format of the mythtv job. You will run mythicalLibrarian --undo, then MythTranscode all files, then mythicalLibrarian --scan mp4 /path_to_my/recordings

The script to transcode all of your recordings would look like this
Code:
#! /bin/bash
mkfifo NamedPipe  #make a pipe
ls -1 >./NamedPipe & #write file list to pipe, one file per line
while read line   #loop through pipe, line by line
do
if [ ${line:(-3)}  = "mpg" ]; then  #if last 3 chars are mpg
   MythTranscode blahBlah $line #$line will be the name of the file  You need to figure out the options for mythTranscode
fi
done <./NamedPipe  #input for the pipe
rm ./NamedPipe  #cleanup the left-over named pipe so it does not clutter the filesystem
echo "All Operations Complete [url=file://\\n]\\n[/url] Done." #this is so you don't stop it prematurely
It'll probly take about a day or two to finish if you have a large library.

I wrote this without a terminal to check, so you may need to debug it.


- GregoryK - 2011-01-02

Adam:

Thanks for the draft shell script. One problem (my own fault, not the fault of mythicalLibrarian) is that running --undo on my current setup won't actually unwind all of the renamed/symlinked files. (This is I think due to manually running --scan as a different user when I was first installing/configuring mythicalLibrarian on my system.) Angry

So your proposed technique won't entirely work around my problem, but maybe if I ran --undo and then put a check in the shell script that made sure not to run mythtranscode on a symlinked file, but only an actual recording file, I could get myself at least part of the way there?

Happy New Year!

/Greg


- rmikulec - 2011-01-04

outleradam Wrote:GregoryK: The problem is XBMC's handling of MPEG2-TS variable framerate. The show comes in at 24fps, and some commercials are appended into the broadcast at 60fps. Using MythTranscode on each file will patch the problem in both MythTV and XBMC by setting it at a fixed framerate. MythTV handles variable framerate with a table which is generated by it's comskip handler. If the file is variable framerate, the frame numbers in the comskip.txt are correct, but XBMC blindly assumes fixed framerate based on the first few frames of the recording.

The patch is to have MythTV transcode every file. This creates a fixed framerate for every show. This should be done before mythicalLibrarian runs. So, set MythTV to run only 1 job at a time. Set MythTV to do MythTranscode job, select MythCommFlag checkbox, and have mythicalLibrarian run last.

Let me know if there are any problems

I assume when you do this you have to remove the JobSuccesful file in order to prevent double transcoding. Is that correct?


- outleradam - 2011-01-04

rmikulec Wrote:I assume when you do this you have to remove the JobSuccesful file in order to prevent double transcoding. Is that correct?
No, you simply put it into the JobSucessful file. It will be implemented on the next mythicalSetup.

GregoryK: You can undo as every user. Each user has his/her own undo file which will revert the files to their original state.


- rmikulec - 2011-01-04

outleradam Wrote:No, you simply put it into the JobSucessful file. It will be implemented on the next mythicalSetup.

Here is my JobSucessful.

PHP Code:
#Run a transcoding program on the file
ffmpeg -i $MoveDir/$ShowFileName.$OriginalExt -target ntsc-vcd $MoveDir/$ShowFileName.$OriginalExt

#Write the new file extension over the old extension for logging
originalext=mp4 

#Run logging for the new file extension so mythicalLibrarian can keep track of files it created.
performLoggingForMoveMode #or performLoggingForLinkMode
 
#remove the mythicalLibrarian symlink
rm "$InputPath"

#make a new symlink
ln -s  "$MoveDir/$ShowFileName.originalext" "$InputPath"

#Enable XBMC Communications which were disabled so mythicalLibrarian did not communicate when the command was called before
XBMCUpdate=Enabled
XBMCClean
=Enabled
XBMCNotify
=Enabled

#Tell mythicalLibrarian to send a message to XBMC and update the library
XBMCAutomate

#Don't exit, just let mythicalLibrarian take back over. 

What would I have to change? I assume it is just the first section.

Any help would be greatly appreciated. Besides the commercials issues it has been working awesome.


- rmikulec - 2011-01-04

If I used this as my JobSucessful would it work?

PHP Code:
#!/bin/sh
VIDEODIR=$1
FILENAME
=$2
# Sanity checking, to make sure everything is in order.
if [ -"$VIDEODIR--"$FILENAME]; then
        
echo "Usage: $0 <VideoDirectory> <FileName>"
        
exit 5
fi
if [ ! -"$VIDEODIR/$FILENAME]; then
        
echo "File does not exist: $VIDEODIR/$FILENAME"
        
exit 6
fi
# The meat of the script. Flag commercials, copy the flagged commercials to
# the cutlist, and transcode the video to remove the commercials from the
# file.
mythcommflag -f $VIDEODIR/$FILENAME
ERROR
=$?
if [ 
$ERROR -gt 126 ]; then
        
echo "Commercial flagging failed for ${FILENAME} with error $ERROR"
        
exit $ERROR
fi
mythcommflag 
--gencutlist -f $VIDEODIR/$FILENAME
ERROR
=$?
if [ 
$ERROR -ne 0 ]; then
        
echo "Copying cutlist failed for ${FILENAME} with error $ERROR"
        
exit $ERROR
fi
mythtranscode 
--honorcutlist --showprogress -i $VIDEODIR/$FILENAME -o $VIDEODIR/$FILENAME.tmp
ERROR
=$?
if [ 
$ERROR -ne 0 ]; then
        
echo "Transcoding failed for ${FILENAME} with error $ERROR"
        
exit $ERROR
fi
mv $VIDEODIR
/$FILENAME $VIDEODIR/$FILENAME.old
mv $VIDEODIR
/$FILENAME.tmp $VIDEODIR/$FILENAME
mythcommflag 
-f $VIDEODIR/${FILENAME} --rebuild
ERROR
=$?
if [ 
$ERROR -ne 0 ]; then
        
echo "Rebuilding seek list failed for ${FILENAME} with error $ERROR"
        
exit $ERROR
fi
mythcommflag 
--clearcutlist -f $VIDEODIR/$FILENAME
ERROR
=$?
if [ 
$ERROR -eq 0 ]; then
        
# Fix the database entry for the file
        
cat << EOF mysql mythconverg
UPDATE 
        recorded
SET
        cutlist 
0,
        
filesize = $(ls -l $VIDEODIR/$FILENAME awk '{print $5}'
WHERE
        basename 
'$FILENAME';
EOF
        
exit 0
else
        echo 
"Clearing cutlist failed for ${FILENAME} with error $ERROR"
        
rm -f $VIDEODIR/$FILENAME.tmp
        
exit $ERROR
fi 



- GregoryK - 2011-01-04

outleradam Wrote:No, you simply put it into the JobSucessful file. It will be implemented on the next mythicalSetup.

GregoryK: You can undo as every user. Each user has his/her own undo file which will revert the files to their original state.

Adam: I realize that there are multiple undos, but somewhere along the way I must have deleted some undo data. I think I have worked out a way to return the files to their original using your undo.sh as a template.

oh, and ... what's a JobSucessful file and where is it located? Apologies for my ignorance on this one, but 'locate' didn't find it on my system.

/Greg


- outleradam - 2011-01-04

Is this thread out of sync or is it just me? I'm just now seeing posts which did not show up before.

GregoryK: The JobSucessful is used for running commands after mythicalLibrarian runs. It is there to extend functionality while utilizing internally generated values... this way you can work with things like Season, Episode, Title, OriginalAirdate, and other values.

rmikulek: it would probly be better to run 1 job at a time and have that run before mythicalLibrarian.


- GregoryK - 2011-01-04

Adam:

Thanks for the explanation, I found the UserJobs wiki page which I hadn't seen before, that helps.

But you think it would still be better to --undo and then run a transcode, and then re-run mythicalLibrarian on the transcoded files, rather than run mythicalLibrarian with a JobSucessful file that transcodes?

My efforts to get the transcoding done are complicated by the recently-discussed audio bug with mythtranscode breaking 5.1 audio in mythtv 0.24. So I am considering using Handbrake to do the transcoding, even though I would prefer to do a mythtranscode with the --mpeg2 flag to save time (storage space is not really an issue in my setup at this point.)

Thanks again for the help & expertise.

/Greg