mythicalLibrarian - a tool to build Movie and TV Show library from MythTV recordings

ubuntite · 2013-03-07 22:24

(2013-03-06 08:56)ubuntite Wrote:
(2013-01-19 08:17)outleradam Wrote: The Symlink=Disabled is kinda a depricated feature. There is a script called MythDataDeleter. http://www.mythtv.org/wiki/MythDataDeleter We wrote that right after mythicalLibrarian switched to MythDataGrabber. MythDataDeleter will do what you want.

So, Symlink=Disabled doesn't seem to be deleting my database entries.

How do I go about implementing MythDataDeleter?

In case this might help someone, this is what I did.

My goal was to:

move a recorded show to my NAS
transcode it to an x264 .mp4 & delete the .mpg
delete the related .txt and .nfo files on the NAS that get generated for the show
delete the show from the myth database
delete the show's thumbnails from the myth recordings directory

The first this I did was to make sure to set Symlink=Move in mythicalLibrarian.

Next I copied this code into /usr/local/bin/MythDataDeleter and did a sudo chmod 755 on it.

Then I inserted the following code into mythicalLibrarian directly underneath "#Insert Custom User Job here" within the "#Successful Completion of mythicalLibrarian" section of the "### USER JOBS ###" section.

Code:

ffmpeg -i "$MoveDir/$ShowFileName.$OriginalExt" -vcodec libx264 -vpre veryfast -deinterlace "$MoveDir/$ShowFileName.mp4"

rm "$MoveDir/$ShowFileName.$OriginalExt"

rm "$MoveDir/$ShowFileName".txt

rm "$MoveDir/$ShowFileName".nfo

/usr/local/bin/MythDataDeleter --filename="$InputFileBasename" --DBPin="$DBPin" --DBHostName="$DBHostName" --DBName="$MySQLMythDb" --DBUserName="$MySQLuser" --DBPassword="$MySQLpass"

rm "$originaldirname/$FileBaseName".*

The next thing I would like to accomplish is to automatically remove the commercial breaks when I do the transcoding. Can anyone give me some assistance? Thanks.

GregoryK · 2013-03-07 22:38

(2013-03-07 22:24)ubuntite Wrote: The next thing I would like to accomplish is to automatically remove the commercial breaks when I do the transcoding. Can anyone give me some assistance? Thanks.

Thanks for posting that ubuntite, very interesting.

For commercial removal, mythtranscode worked great for me using a version of this User Job script in 0.25, but was broken by the 0.26 version of mythtranscode (see this bug).

Other people have been using Lossless Cut in 0.26 to do the cutting and exporting, but it reportedly has some issues with HDHomeRun recordings, and since that is what I use for a tuner, I haven't tried to configure Lossless Cut on my system.

Since you seem to know what you are doing more than I do, maybe one of those two options will work for you.

/Greg

ubuntite · 2013-03-09 09:32

(2013-03-07 22:38)GregoryK Wrote:
(2013-03-07 22:24)ubuntite Wrote: The next thing I would like to accomplish is to automatically remove the commercial breaks when I do the transcoding. Can anyone give me some assistance? Thanks.

Thanks for posting that ubuntite, very interesting.

For commercial removal, mythtranscode worked great for me using a version of this User Job script in 0.25, but was broken by the 0.26 version of mythtranscode (see this bug).

Other people have been using Lossless Cut in 0.26 to do the cutting and exporting, but it reportedly has some issues with HDHomeRun recordings, and since that is what I use for a tuner, I haven't tried to configure Lossless Cut on my system.

Since you seem to know what you are doing more than I do, maybe one of those two options will work for you.

/Greg

I briefly tried removecommercials, but didn't have any luck. I took into consideration the syntax changes of mythcommflag and mythtranscode (-c to --chanid and -s to --starttime), and used mythutil instead of mythcommflag where appropriate.

Anyone have removecommercials working with 0.25?

divingmule · 2013-03-09 15:48

I made a few changes to the RemoveCommercials script, I'll share it here for a while -> http://ubuntuone.com/5VVhAYIp9V0M24yMwxGZPx

I had trouble with the original script not playing nice if running multiple user jobs, so I added nice-ness to the process. I also changed it so that the original file is overwritten instead of being renamed .old. Maybe some other minor changes, can't remember, it's been a while.

I run this script as a MythTV userjob before mythicalLibrarian runs.

Code:

/path/to/commercialremover.sh %DIR% %FILE% %CHANID% %STARTTIME%

I do commercial flagging when the recording starts so, if you want to use this script to run it you will have to un-comment the commercial flagging lines.

On my hardware it takes about five minuets for a thirty minuet recording to complete and file size is reduced by about a third.

As GregoryK said MythTV 0.26 broke the mythtranscode Angry

that was a deal breaker for me, sticking with 0.25.

ubuntite · 2013-03-10 02:09

(2013-03-09 15:48)divingmule Wrote: I made a few changes to the RemoveCommercials script, I'll share it here for a while -> http://ubuntuone.com/5VVhAYIp9V0M24yMwxGZPx

I had trouble with the original script not playing nice if running multiple user jobs, so I added nice-ness to the process. I also changed it so that the original file is overwritten instead of being renamed .old. Maybe some other minor changes, can't remember, it's been a while.

I run this script as a MythTV userjob before mythicalLibrarian runs.

Code:

/path/to/commercialremover.sh %DIR% %FILE% %CHANID% %STARTTIME%

I do commercial flagging when the recording starts so, if you want to use this script to run it you will have to un-comment the commercial flagging lines.

On my hardware it takes about five minuets for a thirty minuet recording to complete and file size is reduced by about a third.

As GregoryK said MythTV 0.26 broke the mythtranscode that was a deal breaker for me, sticking with 0.25.

Thanks for this. I was excited to try your script to see if I would have more luck than with the version I was using. I more closely followed the action this time and noticed a very strange thing. The "Flag Commercials" job that I set to run when the recording starts actually ended before the recording finished.

I got a bunch of these errors:

Code:

Decoder ringbuffer.cpp:1023 (WaitForReadsAllowed) RingBuf(/var/lib/mythtv/recordings/1003_20130309141500.mpg): Taking too long to be allowed to read..

followed by bunch of these:

Code:

Decoder avformatdecoder.cpp:4396 (GetFrame) decoding error#012#011#011#011eno: Broken pipe (32)

I am running my myth-backend on a low-powered nettop with a bunch of other servers running in the backend. Could it be that I am running out of horsepower? Can I tune mythcommflag to be more patient?

divingmule · 2013-03-10 16:00

I don't know, one might assume so :-) do you run the ffmpeg transcode on the same hardware? Maybe try setting the flagging to run after the recording ends or run it from the script.

ubuntite · 2013-03-12 02:57

(2013-03-10 16:00)divingmule Wrote: I don't know, one might assume so :-) do you run the ffmpeg transcode on the same hardware? Maybe try setting the flagging to run after the recording ends or run it from the script.

Hmm... Doesn't seem to matter how I run mythcommflag, it fails. I tried running it after a recording, and I tried running it from the script, but each time it fails with a "decoding error" and a "broken pipe (32)" error.

Obviously this is a mythtv issue. I guess an update somewhere along the way broke mythcommflag. Anyone having similar issues?

ubuntite · 2013-03-14 08:14

Let me respond to my own posts and please forgive me for straying way off topic. I know this has nothing, or at most very little, to do with mythicalLibrarian. The commercialremover script is the user job that I run immediately before mythicalLibrarian.

My problems with flagging commercials seem to be due to either a non-existant or a corrupt seektable. It seems that because I am running a low-powered nettop with a bunch of services and servers, creation of the seektable may not be reliable.

My solution is to make the following changes to the commercialremover script. I added

Code:

mythcommflag --chanid $CHAN --starttime $START --rebuild

(which (re)builds the seektable) immediately before

Code:

mythcommflag --chanid $CHAN --starttime $START --method 7

(which actually flags commercials) in the commercialremover script . I do not flag commercials while recording. You'll note that I explicitly state the commercial flagging method (7=all), so as not to have to rely on this parameter being correctly set within the mythbackend setup.

Next, to get the commercialremover script running properly under myth 0.25, I had to make a change to the line responsible for transcoding, from

Code:

mythtranscode --honorcutlist --showprogress -i $VIDEODIR/$FILENAME -o $VIDEODIR/$FILENAME.tmp

to

Code:

mythtranscode --chanid $CHAN --starttime $START --mpeg2 --honorcutlist --showprogress --outfile $VIDEODIR/$FILENAME.tmp

Not only did I need to change how the recording was passed to the script, but it turned out to be important to specify the "--mpeg2" parameter (which initiates a lossless transcode) if you hadn't previously specified a transcode profile in the myth backed (which I hadn't).

I hope this helps someone.

ubuntite · 2013-03-19 04:28

It looks like an episode with a ":" (colon) in its name causes mythicalLibrarian to crap out when trying to move the file.

Is this a known problem? Is there a fix?

divingmule Posting Freak Posts: 1,278 Joined: Oct 2008 Reputation: 54	2013-03-10 16:00 Post: #956 I don't know, one might assume so :-) do you run the ffmpeg transcode on the same hardware? Maybe try setting the flagging to run after the recording ends or run it from the script.
	find quote

ubuntite Member Posts: 65 Joined: Apr 2008 Reputation: 0	2013-03-19 04:28 Post: #959 It looks like an episode with a ":" (colon) in its name causes mythicalLibrarian to crap out when trying to move the file. Is this a known problem? Is there a fix?
	find quote