A Video Scraper Addon for Humax Foxsat DVR files - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32) +--- Forum: Add-ons (https://forum.kodi.tv/forumdisplay.php?fid=26) +--- Thread: A Video Scraper Addon for Humax Foxsat DVR files (/showthread.php?tid=148107) Pages:
1
2
|
A Video Scraper Addon for Humax Foxsat DVR files - badpenguin - 2012-12-12 I am interested in finding or developing a Video Addon to scrape my video files as copied from a Humax Foxsat DVR. These files look like the following file+directory structure : How I Met Your Mother/ How I Met Your Mother_20121115_1257.hmt How I Met Your Mother_20121115_1257.nts How I Met Your Mother_20121115_1257.ts I want to get the information about these files into the Library. I guess I could use the IMDB video scraper and modify the addon.xml to be able to get the relevant information out of the filenames as above i.e. ignore the "_20121115_1257" and only list the actual video ".ts" files. However the ".hmt" file contains the EPG information about the recording e.g. TV show title and Episode details. I had thought to create my own scraper addon to read these .hmt files to get this info, instead of going to a website such as IMDB. Is making my own ".hmt" addon a sensible idea, or should I just stick with modifying the existing IMDB addon ? If it is a good idea to make this new ".hmt" scraper addon, then are there any similar addons to use as an example ? I've tried searching and found nothing. I believe the existing video scrapers are not python code but rely on the xbmc.metadata.scraper.library, which is written in c++ (in file xbmc\addons\Scraper.cpp). - how can I get my python addon to be called by the VideoInfoScanner.cpp ? - maybe this is only able to call the Scraper.cpp ? - Can my python addon inherit from xbmc.metadata.scraper.library ? I've just noticed that xbmc\addons\Scraper.cpp looks for NFO files which can be used as additional info/hints - perhaps it is better for me to write a program to convert my HMT files to NFO files, and then I can use any existing video scraper ..... Sorry I am new to xbmc, and python coding. Any help gratefully received. Thanks RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2016-09-07 Hi, Resurrecting an ancient thread here but I've spent the last couple of days tackling this problem and have come up with a solution. I'm using a Humax Foxsat with Raydon's custom firmware. I've written a shell script which runs on the Humax. This can be triggered manually or automatically via a cron task (you can install crond using the Raydon webif). The script should be placed in the /media/Video/ folder, and when triggered it scans the drive for .hmt files. It then produces .nfo files for each .hmt and also generates a correctly named symlink for the corresponding .ts file. These files are placed in a separate folder called "KodiTV", under a subfolder created under the title name of each show. Each time the script is triggered, it deletes the KodiTV folder and re-creates and re-populates it. This is so that changes on the Humax (i.e. series deleted on the Humax) are reflected in the target folder. You can then add/scan this "KodiTV" folder in Kodi, or the subfolders on a case-by-case basis as I have. Using TheTVDB scraper ensures that the top-level series name metadata is usually scraped ok from online info, and season / episode numbers are read from the generated .nfo file. The season / episode numbers are derived from the record date stamps that the Humax puts on its files. I did investigate doing something more fancy and attempting to scrape correctly using "airdate" as the hook, but different series have different airdates around the world so to me it was a can of worms. Season is read as YYYYMM and episode is DD. So, it's quite rudimentary but works ok. Please excuse the amateurish scripting work - I am a beginner. I'm sure there are better or more efficient or elegant ways of achieving what I have done but it works ok for me at the moment. Feel free to change things around as you see fit. Code: #!/bin/bash RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2016-09-18 Updated.... Episode number is now DDHHMM (day, hour, minute) to account for multiple episodes being recorded in one day. Episode synopsis is now added to the .nfo via Raydon's hmt tool. Optional Kodi wake and library housekeeping task performed at the end. "Curl" command required for this. General tidy up job of the script and variables. Code: #!/bin/bash RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2017-11-12 "Series" and "Oneoffs" now separated into different directories - rudimentary way to separate ongoing series and movies / oneoff documentaries etc. Encrypted content now skipped. Filenames and nfo "title" field now read from the "title" info contained in the hmt file rather than being derived from the original filename. Target directory is now hidden in the Humax GUI (.KodiTV) Code: #!/bin/bash RE: A Video Scraper Addon for Humax Foxsat DVR files - SlyBadger - 2018-03-04 Please can you post a set of line by line instructions on how to set this up for a complete novice. I have been running the foxsat hdr box for a few years and using it to watch programmes on a tablet very successfully. I am currently trying to experiment with Amazon-Alexa Kodi integration and have so far got things working for my music library but would like to add my foxsat tv shows and movies too. RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2018-03-06 Ok, exactly how much of a novice - are you able to FTP into your Foxsat, or mount it as a SMB share? Do you know how to SSH into your Foxsat? Are you on a Mac or PC? Can you access the the Foxsat web interface installed by Raydon's custom firmware? RE: A Video Scraper Addon for Humax Foxsat DVR files - SlyBadger - 2018-03-06 I have not tried to connect to the box with ftp or SSH. I think I must have set up a samba share originally as the samba package is installed and I can view and access "foxsat" in the windows network and copy off files. Twonkymedia is also running and so I can see and copy files using my web browser. My preference is to use a windows PC although I have raspberry pi on the network running linux so if it's easier to use that I can do. I can access the web interface and have successfully logged into the box to set things to record remotely over the Internet when on holiday. I would be interested to try and set this up if you are able to help. Simon. RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2018-03-07 Ok, Copy the text from the latest version of the script that I posted above and paste into a text editor - on Windows, probably Notepad? Save it on your desktop as "script.sh" Navigate to your Foxsat over SMB and copy the file "script.sh" to media/Video/ Go to the web interface on the Foxsat, click on package management and install "crond", "dropbear" and "hmt". This is where you will now need to log into your Foxsat using SSH. Not done this on Windows but I've heard that people use an application called Putty which is a SSH client I think. Alternatively you could try using an iPad or similar - there are a few SSH clients on the App Store. Log into your foxsat - Code: ssh foxsat.ip.address.here -l root When you're in try..... Code: sh /media/Video/script.sh Then type... Code: ls /media/Video/.KodiTV/Series/ Let me know how you get on up to this point. We can do the tidying up after that. RE: A Video Scraper Addon for Humax Foxsat DVR files - SlyBadger - 2018-06-23 Sorry it's taken a while to reply. When I run the script I get various errors. I've tried commenting out all the lines and bringing them in one by one but struggling to get anywhere. Foxsat-HDR~# sh /media/Video/script.sh ': No such file or directory : No such file or directory ': No such file or directory sh: 2: unknown operand ': No such file or directory ln: /Video: Read-only file system ': No such file or directory ln: /Video: Read-only file system ': No such file or directory ': No such file or directory ': No such file or directory File operations completed RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2018-06-23 Hmmm... I haven't done any of this stuff for a while, so it's difficult to figure out. My Humax box has been running this script twice a day for a year or so without much issue. Try Code: sh -x /media/Video/script.sh You'll get a ton of output but it may help to figure out what the issue is. Paste it here, or in a PM to me if you like. RE: A Video Scraper Addon for Humax Foxsat DVR files - SlyBadger - 2018-07-06 Here's a bit more information: Foxsat-HDR~# sh -x /media/Video/script.sh + ': No such file or directory + cd /media/Video : No such file or directory + counter=0 + ': No such file or directory + [ -lt 2 ] sh: 2: unknown operand + then ': No such file or directory + ln -s -f /media/Video/ / ln: /Video: Read-only file system + else ': No such file or directory + ln -s -f /media/Video// / ln: /Video: Read-only file system + fi ': No such file or directory + done ': No such file or directory + ': No such file or directory + echo File operations completed File operations completed RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2018-07-06 Not really sure what's happening here. SSH into your Foxsat and do Code: ls /media/Video Paste your output here. It looks like the script can't see your media/Video folder, but it doesn't make sense as you can launch the script. Possibly your cutting and pasting of the script code has introduced some weirdness. Do the above and we'll take it from there. RE: A Video Scraper Addon for Humax Foxsat DVR files - dave999 - 2020-09-10 Hi first post so i apologies if i'm supposed to introduce myself elsewhere, but discovered this page due to the info above and wanted to reply with the small bit of progress i have made yes i know its an old thread... I have a Humax HDR Fox T2 which is different from the box the original poster, mwpmorris is using, I suspect that the STB that slybadger is using is also different, and has a media/My Video file structure rather then media/video the foxsat DVR has data in media/video the HDR t2 has its data in media/My Video (Keep in mind that i know little about Unix/linux os so this is just trial and lots of error work) which means the script above, Version 3 . will not work for us all references to media/video where Video is not followed by any other character need to be changed to media/'My Video' in the script as far as i can ascertain if the ref to My Video is in a path statement i.e is terminated in some way with /somedirectory it seems to work so they can be left as media/My Video/some directory or I've just been lucky but you can't cd media/My Video without it saying i can't find My you have to cd media/'My Video' i'd suggest that the script should be cut n pasted form above direct into vi which you have already opened on the set top box vi script.sh and you get an empty file waiting for the script. You use Insert Esc : i in vi to make the changes....you avoid the windows return characters messing up the script Esc:x to exit and save once changed the script will work for all recordings where the HMT file data matches the original series name and the ts and hmt are in the original directory provided the feeview broadcaster followed the standards and didn't mess up the data for example if the Humax box decided to put TOTP1986 recordings in a directory called TOTP1986 the script will create an nfo and a shortcut/symbolic link (to the orginal .ts) named the same as the nfo in the. kodi directory created by the script. it appears to use the hmt data to get the directory name. if however the BBC broadcast TOTP1987 as a continuation to the 1986 series and do not swap some pertinent metatdata somewhere in one of the side car files the the 1987 video is in TOTP1986 directory. in this case an nfo is built but the symbolic link to the .ts is missing, and all i can assume is, that this is because the script went looking for a TOTP1987 directory in order to create the symbolic link and didn't find one, the script is assuming TOTP1987 is the directory in the routine that makes the symbolic link because of something in the HMT the same occurs if you use the STB functionality to move your video data. i.e all films still in their original directory get an nfo and a symbolic link in the .kodi directory, but if you do as i do and separate kids films into their own area, in a directory called Kids films. the script again goes looking for the film in its original directory as it tries to make its short cut/symbolic link which doesn't exist in the root on My Video anymore its in My Video/Kids Films/ so all you get is a nfo in the .kodi directory thus you end up with a .kodi directory which you can transfer via FTP to you PC that has some nice directories that when you FTP them across to you server you get the .nfo, and the ftp software will follow the symbolic link with the same name as the .nfo and find the original .ts file, in its original place, potentially called something less complicated than the new .nfo and copy it across. for the other directories you will just find them full of .nfo and the names of the .nfo files are so wildly different from the names give to, in my case TOTP episodes, and are in directories named from the HMT not the current path, that i have not been able to match them up. and ive got 500 gig of this stuff the directory structure in .kodi is defined by HMT data and directory names and it doesn't match the directory structure in any way shape or form, for many of my media files so i'm getting about a 40% success which is great and i thank the original poster for the script that 40% more than i would have managed I just can't work out how to modify it to fit with a STB that has files on it that are either: 1) moved by the owner using legit functions of the UI on the set top box 2) end up in 1 directory due to the data included or mistakes made in metadata when broadcast for example my TOTP1986 directory for some reason now has anything called TOTP by the BBC in it and covers 86/87/88/89 + specials. At some point in the last 2+ years someone at the BBC has made the decision that TOTPs, no matter what year and series is going to have the same metadata in the HMT file for a specific data point, which makes my STB think its still on the 1986 series. then once every couple of months they don't, and a new directioy TOTP 1987 BIG HITs pops up for 1 or 2 episodes and we go back again to 1986 have spent a long time trying to work out how to modify the script above to cater for the fact that the video's current path is very relevant to making this work 100% of the time...the script treats HMT data as gosple and i need it not to .... but i'm stuck can anyone help i can do the odd dos batch but this is beyond me .... my next move is to use the webif sidecar utility in the Custom firmware on my STB to see if it builds an HMT from the video stream data and the directory structure....the script might work then, but the likelihood of making matters worse is high regards Dave RE: A Video Scraper Addon for Humax Foxsat DVR files - dave999 - 2020-09-11 PS this utility works on linux based boxes to read HMT files, if you had a linux based kodi install, I guess its not beyond the realms of possibility to install it and call it from an addon. but i'm way out of my depth on that front https://hummy.tv/forum/threads/extract-program-metadata-from-downloaded-hmt-file.4953/#post-63087 RE: A Video Scraper Addon for Humax Foxsat DVR files - mwpmorris - 2020-09-11 (2020-09-10, 23:59)dave999 Wrote: Hi Hi Dave, This is a very old thread. So old that I had to read back over my script to try and remember what I did. I’m sure there are a few dodgy bits in there. I don’t really use the Humax much any more as I’ve moved everything over to Kodi and TVHeadend. From memory though, I remember having similar problems when moving files manually, then the script not being able to keep track of where the original TS files were. On my Foxsat’s custom firmware there was a utility that scanned through the media files and rebuilt the HMT files depending on the location of the media. At least I think that’s what it did. I seem to remember having to run that first on occasions, then re-running my script. I’ll see if I can have a look over the next week to try and jog my memory. Matt. |