Kodi Community Forum

Full Version: tvheadend enhancements (I'm working on)
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5
Hi All,

First off, sorry this is a long post Wink

I'm in the process of working on some enhancements to tvheadend in relation to the EPG and DVR code. The original focus of this was to make use of the better upstream EPG data that I have available to me and also to add (where possible) some general improvements.

I've been talking to Andreas (TVH maintainer) and I'm hopeful that these changes (or at least some of them) will find their way in the main tvh code base.

I put together a doc about my suggestions some time ago:
https://docs.google.com/document/d/1mBAG...XYrg0/edit

It gives a reasonable overview of what I was planning, although some refinements have been made. You can mostly ignore the HTSP stuff I've not made any significant changes at this point everything is kept compatible (so as not to break my XBMC frontend). This part of the work is essentially complete, although I am still making occasional changes, and there is plenty of testing to be done.

One major drawback at the moment, is most of the really cool features don't exist except for a very limited subset of people (namely UK freesat users, and even then you need to request an API key from the upstream data provider). Since this is the only source (I have) for proper structured EPG data. XMLTV simply doesn't cut it (I'm going to be talking to the XMLTV devs as well). However that said some attempt to massage XMLTV data is made and the basic internal structure within tvheadend does make some tasks much easier.

This is particularly true in the DVR code. One of the other reasons to start looking at this is to enable a series link feature in tvheadend and to try and do some cool features around that (like re-scheduling of missed recordings, automatic selection of higher quality stream etc...).

I've also put together a doc about current thoughts on DVR configuration options (really just a brain dump).
https://docs.google.com/a/adamsutton.me....KP_-Y/edit

What I'm really looking for are two things:

1. Any input from the PVR devs (Lars I'm thinking you here Wink, but others too) on whether any of this can be integrated into XBMC. Some simple things like:
- button for series link request (I did start working on this myself but got sidetracked onto the more detailed TVH updates).
- use of some mods I made to recordEvent HTSP method that will allow you to do things properly.
2. Any input from users on what they'd like to see mainly thinking about EPG and DVR stuff here. I'm not a DVB/Video expert, so don't ask for things like time shifting (I'd love to see it too!).

Regards
Adam
Hi,

that are great news ! I'm a fan of tvheadend but I guess / seems to me that development stalled ..

Atm I'm not sure what I'd like to see .. great would be if you could enhance xmltv / web scrapping support to functionality like this:

http://forum.xbmc.org/showthread.php?tid=133913

Means, like seen in XBMC too for video/music related information, a

- generic scrapper API in tvheadend
- that reads plain text files
- containing parsing rules for parsing tv guide holding web sites

to receive enhanced TV information ..

Greetz

LastCoder
I think its fair to say that Andreas is no longer actively developing tvh, however I think others are trying to take on the mantle. My involvement is fairly restricted at this point as I've no understanding of the wider DVB code. However Andreas is definitely open to patches etc.. so we'll see what happens and I know there is definitely some tie in the the work Lars is doing.

Looking at webgrab+plus, it seems to me to be just another XMLTV grabber (with some added metadata from IMDB). Which ultimately suffers from the same basic limitations, i.e the XMLTV format, as other XMLTV scripts. Though I must admit I only skimmed the front page so apologies if I missed something.

My aim, in the fullness of time, is to get people to use a much richer data format (where the upstream content supports it) that allows the various relationships within the EPG data to be properly expressed. I've sent some messages to the XMLTV devs and we'll see what happens.

That being said as it webgrab+plus outputs XMLTV data then it'll still work with tvheadend, my mods make non-standard grabbers a bit easier to integrate because there are now 2 methods of configuration. 1) as existing select a grabber and interval and 2) enable a unix domain socket in to which to pump data. This can be useful for setting up crons to run the scrapers (e.g. I run my scraper hourly for 24 hour download and daily for 14 day download, to get a mix of regular updates and long term info).

As for a "generic" scraper API, not sure that's going to happen given the way tvheadend works. However that being said the use of domain sockets for data input means that anyone can write a scraper as long as it outputs data in one of the support formats. This currently only includes XMLTV and my own PyEPG format and will eventually include a HTSMSG format (the internal JSON-like format used by tvheadend for internal and remote messaging).

Adam
This sounds very interesting Adam,
I currently 'twiddle' about with tvheadend myself and have done a few modifications to xmltv and in particular the EIT grabber to try and get myself a decent 'as-full-as-I-can' EPG that works, at the moment it appears to be working fine and keeping my EPG topped up, but I know this isn't a perfect solution.

One thing I'm a little unclear about from your docs, etc, is that where you are getting the EPG from, are you getting it from the EIT or from a 3rd party source (I notice the API key so I'm asusming the atlas API is part of this mashup?

If you're after users/devs to try stuff out then sign me up as I'm constantly tweaking about with the setup I use here (GIT compiled xbmc front-end and same for tvh backend at present, dual freesat receivers + one dvb-t tuner)
WARNING: I type long winded answers Wink

Hey Andy,

yeah testers are VERY welcome. I'm sorta getting to that point where its nearly there (not complete, but useable) but I have a) some bugs to fix and b) some questions that I kind answer on my own about directions to take etc... Hence the reason to I'm beginning to shout about this a bit more widely (thus far limited to #hts).

If you've improved the EIT grabber I'd love to see what you've done. This was on my list of things to do (to get extra metadata etc..) but I've not got there yet (I don't use it myself).

Much of the "inspiration" about how best to structure things indeed came from Atlas system. Basically I got into the whole area because the XMLTV UK scraper script was awful (in terms of performance) I spent a few hours re-writing it in python and sped it up 30-60 fold (some dodgy string/date processing). And I've been hacking around ever since.

Basically I started by writing my own competitor (really I don't see it as a competitor merely a demonstration of a better way Wink as I'm now trying to engage the XMLTV devs) to XMLTV format. This uses a more structured output that groups things into brands (shows), series (seasons), episodes and broadcasts (specific airings of an episode). This is all very easy if you get your data from a service like atlas which maintains a properly linked database. I wrote a python script that does the fetching and output (it'll do XMLTV format if you really want). Code is @ https://github.com/adamsutton/PyEPG. Unfortunately there are 2 major drawbacks to this 1) its UK freesat specific at the moment (extending to ALL UK wouldn't be difficult but I'm a freesat user) and 2) to use it you need an API key for atlas. Metabroadcast (the guys that maintain atlas) are trying to get this changed but ultimately they get their data from PA and its not currently free (watch this space, though I've been watching a while). But I think they're open to giving out keys at the moment.

I've then updated tvheadend to make use of a similar structure internally, thus making it possible to import the output of my script. And although XMLTV doesn't fit perfectly (since its a very flat structure) you can somewhat "massage" it to fit. Which has also been done.

So basically I've re-written the entire internal EPG data structures for tvheadend, added a "somewhat" modular grabber framework, added a new grab module for my script, updated the xmltv grabber and the eit code. Some attempt to merge EIT and non-EIT is done, however I strongly recommend turning EIT off if you're using my script. You run the risk of losing all the benefits (when it can't match things and the EIT data stomps all over the properly linked stuff).

You can still configure TVH as before, select a grabber and period. But you can also enable some Unix domain sockets, so that you can dump the data from externally run scripts (i.e. cron jobs) to give more flexibility.

As you can probably see there are still some hurdles to overcome, but I'm making good progress and so getting people on board is important to keep momentum up (and stopping me getting bored). One thing I should note, if you do want to try things a) its unstable and b) there's no migration for the existing epgdb file format (so you'll have to start from scratch).

My branch is at https://github.com/adamsutton/tvheadend/...pg-rewrite so feel free to poke around.

And if you want to discuss things in more detail mail me direct at dev @ adamsutton me uk or pop along to #hts (I try to be about as much as I can).

Regards
Adam
adamsutton Wrote:Looking at webgrab+plus, it seems to me to be just another XMLTV grabber (with some added metadata from IMDB). Which ultimately suffers from the same basic limitations, i.e the XMLTV format, as other XMLTV scripts. Though I must admit I only skimmed the front page so apologies if I missed something.

The difference is - its expandable through parsing rules so many many sites have been added. There is no other grabber that will do that atm.

I'm not sure if it makes sence to enhance XMLTV as long as the format is not fully utilized .. I would be very happy to have all the information that could be shown with XMLTV for every channel and every program .. so if you could manage this, you can start thinking about another formats holding more information ..

But happy to see people work on tvheadend ;-)

Greetz

LastCoder
But... it's still just another XMLTV scraper Wink so it should work with tvheadend now and in the future (though you might need to put a wrapper script around it at the moment because of how tvh detects xmltv scripts, my mods remove that need since you can drive it how you like).

Also its worth pointing at that generally "web" based scrapers are far from ideal (though plenty of good ones exist). Lots of reasons for that. Its much better, where possible, to use proper EPG data services such as Atlas.

I do intend, time permitting, to include some extra metadata in the internal TVH EPG database, however you've still got to convince the front-end guys (in this case Lars et al) to make use of this extra data in some way. But I accept that getting it there in the first place is definitely a good starting point.

However my point still stands that XMLTV simply doesn't cut it. It lacks the ability to properly express how items within the EPG are linked together, this is important for a variety of reasons (though of course you can get by without it as people have until now). One of the current features missing from TVH (and XBMC for that matter) is series linking. This can currently be achieved in TVH using an autorec with a regexp title match, but its less than perfect. I want something like on commercial PVR boxes (else it'll fail the WAT).

My approach is more similar to that used in say a Sky Box (I'm making some assumptions here, I don't actually have knowledge of how they do it, but I know they use the PA data in at least some respect as its used on their online guide). Because things are fully linked features such as series linking (something I really need and have added) are trivial, I don't have to do horrible regexp matching of titles because I implicitly know from the structure which entries in the EPG are linked together.

The links also have benefits for things like cross channel linking (for example something like Euro 2012, which is on both BBC and ITV, in the PA data ALL matches are part of the same brand so linking the LOT would be trivial), duplicate detection (episodes have unique IDs, so I know implicitly whether I'm already record/have recorded a given episode), re-scheduling (again if an event gets missed I know implicitly which episode it was and all the other locations of that episode in the schedule so I can automatically re-schedule it, even on another channel, without a massive search).

That being said, its obviously not quite as simple as I make it out as there are various ways in which different people will wants things to work and adding in the multitude of configuration items has generally been the hardest point. But doing something akin to a sky box (which is actually quite dumb) is trivial.

There are definitely a few of us still keeping TVH alive and adding new features Smile
there are some other people working on improving/rewriting tvheadend, but they might not have published details about it yet. ping me on IRC (nick = opdenkamp) please
For many services all the data for a 5-7 (or more) EPG is right there in the feed, no need for XMLTV, api keys, charges, whatever. This is what VDR's EEPG plugin does (or tries to), it supports various services (because of course we can't possibly have all services using the same methods ...) such as Freesat's, OpenTV (various Sky services), etc. There are a couple of requests for this functionality in tvheadend - including one to which I replied. The solution can be taken further since the logical channel numbers are often included in the data - thus you could automatically populate the tvheadend channels listing with the channel and its number. You'd just need to select the relevant service in tvheadend. I believe that series link information is contained within the data - I think Mediaportal's EEPG like plugin supports this, or wants to. Not a solution for everyone, everywhere but none the less a good solution for a lot of users.
Hey Lars,

I'm well aware that others are working on it Smile and I believe I recognised your hand in some of the code Wink

I did try to get your attention some months back when I first got interested in doing this sort of thing but you were very busy with getting out the basic Eden stuff so I went away Smile

I'll see if I can get hold of you on IRC.
Drae,

thanks for that I'll have to look into it that. I was aware that a 7 day EIT EPG existed on some services. I wasn't aware that the series link information was available in the stream. But that would be very useful Smile

I'll have a look at that VDR/MediaPortal plugins when I get a chance.

Adam
Drae,

Just to update on my thinking....

I still think that the stuff I have is beneficial, in as much as the data is much better structured and in the event that access to something like Atlas exists you can get way more info than is available in the OTA EPG services. Something like Atlas is also nice as its an open standard, no constant battle to keep up with rev eng proprietary protocols. But I accept the point that its not currently free (though its being worked on) and having something that "just works" out of the box would be great Smile

Moving on...

I've had a look through the various bits of code that decode the proprietary EPG services (there are quite a few, I can't believe it's taken this long for someone to point them out to me or for me to stumble on them). The XMLTV grabber script (tv_grab_dvb_plus) is probably the easiest to understand, but it doesn't appear to have all the features.

I "think" that it should be possible to use/learn from the code (probably from a combination of the existing implementations) and integrate at least "something" into TVH that will handle these protocols. Though I've only just started looking and I've not yet got my existing changes included.

There does appear to be a potentially interim solution (which is marginally easier with my tvh mods) which includes shutting down TVH running the above tv_grab_dvb_plus script (dump to file), start up TVH and dump the XMLTV data back in.

Though this is a "bit" clumsy, does mean stopping TVH (but it could be done overnight) and also wouldn't be able to hold all the information (such as series link) since XMLTV simply doesn't have such a concept.

To sum up...

I think it should be possible to integrate some of the code and get a working TVH solution (without the above kludge), and I'm going to have a look, so watch this space.

Adam
(2012-06-15, 14:43)adamsutton Wrote: [ -> ]I still think that the stuff I have is beneficial, in as much as the data is much better structured and in the event that access to something like Atlas exists you can get way more info than is available in the OTA EPG services. Something like Atlas is also nice as its an open standard, no constant battle to keep up with rev eng proprietary protocols. But I accept the point that its not currently free (though its being worked on) and having something that "just works" out of the box would be great Smile
Yes I think Atlas looks like a nice service and as you say incorporates a lot of additional (scraper like) information. At the end of the day what I'd like to see from the epg is everything currently possible with a sky box (and its equivs) but with the rich data and gui of xbmc. Whatever makes that happen is a big "win" in my book!
(2012-06-15, 14:43)adamsutton Wrote: [ -> ]There does appear to be a potentially interim solution (which is marginally easier with my tvh mods) which includes shutting down TVH running the above tv_grab_dvb_plus script (dump to file), start up TVH and dump the XMLTV data back in.
Yeah, this is actually the only way to currently do it because tvheadend locks the frontends. Indeed one of the responses to the request over at lonelycoder was something like "well you can just dump the data to xmltv and use tv_grab_file so what's the issue?". The issue is that's a real pita workaround - firstly you need to check nothing is using the frontend/s, check nothing is set to use them in the next X seconds, shut down tvh, run the grabber (which never properly worked for me with Sky at least .. had to faff around some iirc inc clean up the xml it produces), restart tvheadend and let the tv_grab_file import it.

Having it properly integrated is obviously a much much nicer solution. It can grab data at startup, it can grab data regularly when no frontends are tuned (making use of a second/third frontend if available and others are in use), it could force grab if necessary, etc. etc. If I had the knowledge it's something I'd have done myself, I did look-see but it's too far out my comfort zone Big Grin
(2012-06-15, 14:43)adamsutton Wrote: [ -> ]I think it should be possible to integrate some of the code and get a working TVH solution (without the above kludge), and I'm going to have a look, so watch this space.
Awesome Smile
OK,

it's a promising start... after hacking away for a few mins to fix the bloody tuning code (what a mess!) in the grabber script I have now got it to dump the sky EPG. This is as a standalone mind you, but the point is:

1) I have a working system that I can use to a) verify things and b) see how to implement in TVH
and
2) I already understand most of the way the parsing code work.

But that still leaves figuring out the best way to integrate it etc... I have some ideas but it's not going to be a 5 min job.

But I'm fairly positive it can be done (and even better I think I might be able to do it Wink ).
I am using Tvheadend the main issue I have is with the EPG. Here in New Zealand the guide is transmitted using MHEG5 over DVB-T and covers 8 days. I dont know where else in the world is using this format but no DVR software seems to support it. Is there a possiblity this could be looked at?
Pages: 1 2 3 4 5