[Linux] tvheadend enhancements (I'm working on) - Printable Version
+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: PVR Development (/forumdisplay.php?fid=136)
+--- Thread: [Linux] tvheadend enhancements (I'm working on) (/showthread.php?tid=133917)
tvheadend enhancements (I'm working on) - adamsutton - 2012-06-13 11:52
First off, sorry this is a long post
I'm in the process of working on some enhancements to tvheadend in relation to the EPG and DVR code. The original focus of this was to make use of the better upstream EPG data that I have available to me and also to add (where possible) some general improvements.
I've been talking to Andreas (TVH maintainer) and I'm hopeful that these changes (or at least some of them) will find their way in the main tvh code base.
I put together a doc about my suggestions some time ago:
It gives a reasonable overview of what I was planning, although some refinements have been made. You can mostly ignore the HTSP stuff I've not made any significant changes at this point everything is kept compatible (so as not to break my XBMC frontend). This part of the work is essentially complete, although I am still making occasional changes, and there is plenty of testing to be done.
One major drawback at the moment, is most of the really cool features don't exist except for a very limited subset of people (namely UK freesat users, and even then you need to request an API key from the upstream data provider). Since this is the only source (I have) for proper structured EPG data. XMLTV simply doesn't cut it (I'm going to be talking to the XMLTV devs as well). However that said some attempt to massage XMLTV data is made and the basic internal structure within tvheadend does make some tasks much easier.
This is particularly true in the DVR code. One of the other reasons to start looking at this is to enable a series link feature in tvheadend and to try and do some cool features around that (like re-scheduling of missed recordings, automatic selection of higher quality stream etc...).
I've also put together a doc about current thoughts on DVR configuration options (really just a brain dump).
What I'm really looking for are two things:
1. Any input from the PVR devs (Lars I'm thinking you here , but others too) on whether any of this can be integrated into XBMC. Some simple things like:
- button for series link request (I did start working on this myself but got sidetracked onto the more detailed TVH updates).
- use of some mods I made to recordEvent HTSP method that will allow you to do things properly.
2. Any input from users on what they'd like to see mainly thinking about EPG and DVR stuff here. I'm not a DVB/Video expert, so don't ask for things like time shifting (I'd love to see it too!).
RE: tvheadend enhancements (I'm working on) - LastCoder - 2012-06-13 14:41
that are great news ! I'm a fan of tvheadend but I guess / seems to me that development stalled ..
Atm I'm not sure what I'd like to see .. great would be if you could enhance xmltv / web scrapping support to functionality like this:
Means, like seen in XBMC too for video/music related information, a
- generic scrapper API in tvheadend
- that reads plain text files
- containing parsing rules for parsing tv guide holding web sites
to receive enhanced TV information ..
RE: tvheadend enhancements (I'm working on) - adamsutton - 2012-06-13 14:55
I think its fair to say that Andreas is no longer actively developing tvh, however I think others are trying to take on the mantle. My involvement is fairly restricted at this point as I've no understanding of the wider DVB code. However Andreas is definitely open to patches etc.. so we'll see what happens and I know there is definitely some tie in the the work Lars is doing.
Looking at webgrab+plus, it seems to me to be just another XMLTV grabber (with some added metadata from IMDB). Which ultimately suffers from the same basic limitations, i.e the XMLTV format, as other XMLTV scripts. Though I must admit I only skimmed the front page so apologies if I missed something.
My aim, in the fullness of time, is to get people to use a much richer data format (where the upstream content supports it) that allows the various relationships within the EPG data to be properly expressed. I've sent some messages to the XMLTV devs and we'll see what happens.
That being said as it webgrab+plus outputs XMLTV data then it'll still work with tvheadend, my mods make non-standard grabbers a bit easier to integrate because there are now 2 methods of configuration. 1) as existing select a grabber and interval and 2) enable a unix domain socket in to which to pump data. This can be useful for setting up crons to run the scrapers (e.g. I run my scraper hourly for 24 hour download and daily for 14 day download, to get a mix of regular updates and long term info).
As for a "generic" scraper API, not sure that's going to happen given the way tvheadend works. However that being said the use of domain sockets for data input means that anyone can write a scraper as long as it outputs data in one of the support formats. This currently only includes XMLTV and my own PyEPG format and will eventually include a HTSMSG format (the internal JSON-like format used by tvheadend for internal and remote messaging).
RE: tvheadend enhancements (I'm working on) - andyb2000 - 2012-06-13 15:48
This sounds very interesting Adam,
I currently 'twiddle' about with tvheadend myself and have done a few modifications to xmltv and in particular the EIT grabber to try and get myself a decent 'as-full-as-I-can' EPG that works, at the moment it appears to be working fine and keeping my EPG topped up, but I know this isn't a perfect solution.
One thing I'm a little unclear about from your docs, etc, is that where you are getting the EPG from, are you getting it from the EIT or from a 3rd party source (I notice the API key so I'm asusming the atlas API is part of this mashup?
If you're after users/devs to try stuff out then sign me up as I'm constantly tweaking about with the setup I use here (GIT compiled xbmc front-end and same for tvh backend at present, dual freesat receivers + one dvb-t tuner)
RE: tvheadend enhancements (I'm working on) - adamsutton - 2012-06-13 17:13
WARNING: I type long winded answers
yeah testers are VERY welcome. I'm sorta getting to that point where its nearly there (not complete, but useable) but I have a) some bugs to fix and b) some questions that I kind answer on my own about directions to take etc... Hence the reason to I'm beginning to shout about this a bit more widely (thus far limited to #hts).
If you've improved the EIT grabber I'd love to see what you've done. This was on my list of things to do (to get extra metadata etc..) but I've not got there yet (I don't use it myself).
Much of the "inspiration" about how best to structure things indeed came from Atlas system. Basically I got into the whole area because the XMLTV UK scraper script was awful (in terms of performance) I spent a few hours re-writing it in python and sped it up 30-60 fold (some dodgy string/date processing). And I've been hacking around ever since.
Basically I started by writing my own competitor (really I don't see it as a competitor merely a demonstration of a better way as I'm now trying to engage the XMLTV devs) to XMLTV format. This uses a more structured output that groups things into brands (shows), series (seasons), episodes and broadcasts (specific airings of an episode). This is all very easy if you get your data from a service like atlas which maintains a properly linked database. I wrote a python script that does the fetching and output (it'll do XMLTV format if you really want). Code is @ https://github.com/adamsutton/PyEPG. Unfortunately there are 2 major drawbacks to this 1) its UK freesat specific at the moment (extending to ALL UK wouldn't be difficult but I'm a freesat user) and 2) to use it you need an API key for atlas. Metabroadcast (the guys that maintain atlas) are trying to get this changed but ultimately they get their data from PA and its not currently free (watch this space, though I've been watching a while). But I think they're open to giving out keys at the moment.
I've then updated tvheadend to make use of a similar structure internally, thus making it possible to import the output of my script. And although XMLTV doesn't fit perfectly (since its a very flat structure) you can somewhat "massage" it to fit. Which has also been done.
So basically I've re-written the entire internal EPG data structures for tvheadend, added a "somewhat" modular grabber framework, added a new grab module for my script, updated the xmltv grabber and the eit code. Some attempt to merge EIT and non-EIT is done, however I strongly recommend turning EIT off if you're using my script. You run the risk of losing all the benefits (when it can't match things and the EIT data stomps all over the properly linked stuff).
You can still configure TVH as before, select a grabber and period. But you can also enable some Unix domain sockets, so that you can dump the data from externally run scripts (i.e. cron jobs) to give more flexibility.
As you can probably see there are still some hurdles to overcome, but I'm making good progress and so getting people on board is important to keep momentum up (and stopping me getting bored). One thing I should note, if you do want to try things a) its unstable and b) there's no migration for the existing epgdb file format (so you'll have to start from scratch).
My branch is at https://github.com/adamsutton/tvheadend/tree/epg-rewrite so feel free to poke around.
And if you want to discuss things in more detail mail me direct at dev @ adamsutton me uk or pop along to #hts (I try to be about as much as I can).
RE: tvheadend enhancements (I'm working on) - LastCoder - 2012-06-14 11:00
adamsutton Wrote:Looking at webgrab+plus, it seems to me to be just another XMLTV grabber (with some added metadata from IMDB). Which ultimately suffers from the same basic limitations, i.e the XMLTV format, as other XMLTV scripts. Though I must admit I only skimmed the front page so apologies if I missed something.
The difference is - its expandable through parsing rules so many many sites have been added. There is no other grabber that will do that atm.
I'm not sure if it makes sence to enhance XMLTV as long as the format is not fully utilized .. I would be very happy to have all the information that could be shown with XMLTV for every channel and every program .. so if you could manage this, you can start thinking about another formats holding more information ..
But happy to see people work on tvheadend ;-)
RE: tvheadend enhancements (I'm working on) - adamsutton - 2012-06-14 15:26
But... it's still just another XMLTV scraper so it should work with tvheadend now and in the future (though you might need to put a wrapper script around it at the moment because of how tvh detects xmltv scripts, my mods remove that need since you can drive it how you like).
Also its worth pointing at that generally "web" based scrapers are far from ideal (though plenty of good ones exist). Lots of reasons for that. Its much better, where possible, to use proper EPG data services such as Atlas.
I do intend, time permitting, to include some extra metadata in the internal TVH EPG database, however you've still got to convince the front-end guys (in this case Lars et al) to make use of this extra data in some way. But I accept that getting it there in the first place is definitely a good starting point.
However my point still stands that XMLTV simply doesn't cut it. It lacks the ability to properly express how items within the EPG are linked together, this is important for a variety of reasons (though of course you can get by without it as people have until now). One of the current features missing from TVH (and XBMC for that matter) is series linking. This can currently be achieved in TVH using an autorec with a regexp title match, but its less than perfect. I want something like on commercial PVR boxes (else it'll fail the WAT).
My approach is more similar to that used in say a Sky Box (I'm making some assumptions here, I don't actually have knowledge of how they do it, but I know they use the PA data in at least some respect as its used on their online guide). Because things are fully linked features such as series linking (something I really need and have added) are trivial, I don't have to do horrible regexp matching of titles because I implicitly know from the structure which entries in the EPG are linked together.
The links also have benefits for things like cross channel linking (for example something like Euro 2012, which is on both BBC and ITV, in the PA data ALL matches are part of the same brand so linking the LOT would be trivial), duplicate detection (episodes have unique IDs, so I know implicitly whether I'm already record/have recorded a given episode), re-scheduling (again if an event gets missed I know implicitly which episode it was and all the other locations of that episode in the schedule so I can automatically re-schedule it, even on another channel, without a massive search).
That being said, its obviously not quite as simple as I make it out as there are various ways in which different people will wants things to work and adding in the multitude of configuration items has generally been the hardest point. But doing something akin to a sky box (which is actually quite dumb) is trivial.
There are definitely a few of us still keeping TVH alive and adding new features
RE: tvheadend enhancements (I'm working on) - opdenkamp - 2012-06-14 15:48
there are some other people working on improving/rewriting tvheadend, but they might not have published details about it yet. ping me on IRC (nick = opdenkamp) please
RE: tvheadend enhancements (I'm working on) - Drae - 2012-06-14 17:02
For many services all the data for a 5-7 (or more) EPG is right there in the feed, no need for XMLTV, api keys, charges, whatever. This is what VDR's EEPG plugin does (or tries to), it supports various services (because of course we can't possibly have all services using the same methods ...) such as Freesat's, OpenTV (various Sky services), etc. There are a couple of requests for this functionality in tvheadend - including one to which I replied. The solution can be taken further since the logical channel numbers are often included in the data - thus you could automatically populate the tvheadend channels listing with the channel and its number. You'd just need to select the relevant service in tvheadend. I believe that series link information is contained within the data - I think Mediaportal's EEPG like plugin supports this, or wants to. Not a solution for everyone, everywhere but none the less a good solution for a lot of users.
RE: tvheadend enhancements (I'm working on) - adamsutton - 2012-06-14 17:04
I'm well aware that others are working on it and I believe I recognised your hand in some of the code
I did try to get your attention some months back when I first got interested in doing this sort of thing but you were very busy with getting out the basic Eden stuff so I went away
I'll see if I can get hold of you on IRC.