Linux tvheadend enhancements (I'm working on)

**adamsutton** · 2012-06-14 21:37

Drae,

thanks for that I'll have to look into it that. I was aware that a 7 day EIT EPG existed on some services. I wasn't aware that the series link information was available in the stream. But that would be very useful Smile

I'll have a look at that VDR/MediaPortal plugins when I get a chance.

Adam

**adamsutton** · 2012-06-15 14:43

Drae,

Just to update on my thinking....

I still think that the stuff I have is beneficial, in as much as the data is much better structured and in the event that access to something like Atlas exists you can get way more info than is available in the OTA EPG services. Something like Atlas is also nice as its an open standard, no constant battle to keep up with rev eng proprietary protocols. But I accept the point that its not currently free (though its being worked on) and having something that "just works" out of the box would be great Smile

Moving on...

I've had a look through the various bits of code that decode the proprietary EPG services (there are quite a few, I can't believe it's taken this long for someone to point them out to me or for me to stumble on them). The XMLTV grabber script (tv_grab_dvb_plus) is probably the easiest to understand, but it doesn't appear to have all the features.

I "think" that it should be possible to use/learn from the code (probably from a combination of the existing implementations) and integrate at least "something" into TVH that will handle these protocols. Though I've only just started looking and I've not yet got my existing changes included.

There does appear to be a potentially interim solution (which is marginally easier with my tvh mods) which includes shutting down TVH running the above tv_grab_dvb_plus script (dump to file), start up TVH and dump the XMLTV data back in.

Though this is a "bit" clumsy, does mean stopping TVH (but it could be done overnight) and also wouldn't be able to hold all the information (such as series link) since XMLTV simply doesn't have such a concept.

To sum up...

I think it should be possible to integrate some of the code and get a working TVH solution (without the above kludge), and I'm going to have a look, so watch this space.

Adam

Drae · 2012-06-15 20:23

(2012-06-15 14:43)adamsutton Wrote: I still think that the stuff I have is beneficial, in as much as the data is much better structured and in the event that access to something like Atlas exists you can get way more info than is available in the OTA EPG services. Something like Atlas is also nice as its an open standard, no constant battle to keep up with rev eng proprietary protocols. But I accept the point that its not currently free (though its being worked on) and having something that "just works" out of the box would be great

Yes I think Atlas looks like a nice service and as you say incorporates a lot of additional (scraper like) information. At the end of the day what I'd like to see from the epg is everything currently possible with a sky box (and its equivs) but with the rich data and gui of xbmc. Whatever makes that happen is a big "win" in my book!

(2012-06-15 14:43)adamsutton Wrote: There does appear to be a potentially interim solution (which is marginally easier with my tvh mods) which includes shutting down TVH running the above tv_grab_dvb_plus script (dump to file), start up TVH and dump the XMLTV data back in.

Yeah, this is actually the only way to currently do it because tvheadend locks the frontends. Indeed one of the responses to the request over at lonelycoder was something like "well you can just dump the data to xmltv and use tv_grab_file so what's the issue?". The issue is that's a real pita workaround - firstly you need to check nothing is using the frontend/s, check nothing is set to use them in the next X seconds, shut down tvh, run the grabber (which never properly worked for me with Sky at least .. had to faff around some iirc inc clean up the xml it produces), restart tvheadend and let the tv_grab_file import it.

Having it properly integrated is obviously a much much nicer solution. It can grab data at startup, it can grab data regularly when no frontends are tuned (making use of a second/third frontend if available and others are in use), it could force grab if necessary, etc. etc. If I had the knowledge it's something I'd have done myself, I did look-see but it's too far out my comfort zone Big Grin

(2012-06-15 14:43)adamsutton Wrote: I think it should be possible to integrate some of the code and get a working TVH solution (without the above kludge), and I'm going to have a look, so watch this space.

Awesome

**adamsutton** · 2012-06-15 22:07

OK,

it's a promising start... after hacking away for a few mins to fix the bloody tuning code (what a mess!) in the grabber script I have now got it to dump the sky EPG. This is as a standalone mind you, but the point is:

1) I have a working system that I can use to a) verify things and b) see how to implement in TVH
and
2) I already understand most of the way the parsing code work.

But that still leaves figuring out the best way to integrate it etc... I have some ideas but it's not going to be a 5 min job.

But I'm fairly positive it can be done (and even better I think I might be able to do it Wink

).

the_mouse · 2012-06-16 01:44

I am using Tvheadend the main issue I have is with the EPG. Here in New Zealand the guide is transmitted using MHEG5 over DVB-T and covers 8 days. I dont know where else in the world is using this format but no DVR software seems to support it. Is there a possiblity this could be looked at?

**adamsutton** · 2012-06-16 11:31

Mouse/Drae,

Just to keep you posted on progress...

I grabbed a raw dump of the data read by the tv_grab script and wrote a quick python script to check I could parse it (just to check my understanding of the protocol was right) looks ok, but I need to tidy it up and make it a little more robust and figure out just how much I can make use of. Then I'll translate to C and look at adding to my tvh branch.

I think I'll try a first pass which will allow "little" configuration, just to make sure I can integrate into TVH ok. I'll look at making it more configurable later.

With regards to your point Mouse there does appear to by MHEG5 support in most of the plugins/scripts I've been using to base the work on. Although my first goal is to get opentv working (since I can actually test that) in time I might be able to write an MHEG5 decoder (it looks fairly straight forward), though I'll need some help testing it as I have no means of doing it myself.

liquidsky · 2012-06-16 19:31

Hi Adam,
I tried your version but tvheadend crashes.
The compilation finish with no error:

Quote:/ configure --release
make
make install

This is the content of syslog:

Code:

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: webui: No source path providing HTTP content: "/static". Checked in "./" and in the binary's embedded file system.

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: webui: No source path providing HTTP content: "/docs". Checked in "./" and in the binary's embedded file system.

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: webui: No source path providing HTTP content: "/docresources". Checked in "./" and in the binary's embedded file system.

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: cwc: Attemping to connect to 127.0.0.1:10020

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: cwc: Attemping to connect to 127.0.0.1:10000

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: cwc: Connected to 127.0.0.1:10000

Jun 16 19:04:28 xbmc-mmb tvheadend[2234]: cwc: Connected to 127.0.0.1:10020

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: Signal: 6 in PRG: /usr/local/bin/tvheadend () [ffe1a5754aba5b0906556eead3cf4a54034bb476] CWD: /

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: Fault address 0x8ba (N/A)

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: Loaded libraries: /lib/i386-linux-gnu/librt.so.1 /lib/i386-linux-gnu/libdl.so.2 /usr/lib/i386-linux-gnu/libavahi-common.so.3 /usr/lib/i386-linux-gnu/libavahi-client.so.3 /lib/libssl.so.0.9.8 /lib/libcrypto.so.0.9.8 /lib/i386-linux-gnu/libpthread.so.0 /lib/i386-linux-gnu/libm.so.6 /lib/i386-linux-gnu/libc.so.6 /lib/ld-linux.so.2 /lib/i386-linux-gnu/libdbus-1.so.3 /lib/i386-linux-gnu/libz.so.1 /lib/i386-linux-gnu/libnss_compat.so.2 /lib/i386-linux-gnu/libnsl.so.1 /lib/i386-linux-gnu/libnss_nis.so.2 /lib/i386-linux-gnu/libnss_files.so.2

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: Register dump [19]: 00000033 c1500000 ffff007b 0000007b 00718ff4 006f269e bfea0108 bfea00fc 000008ba 00000006 000008ba 00000000 00000000 00000000 00371416 00000073 00000202 bfea00fc 0000007b

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: STACKTRACE

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: /usr/src/tvheadend/src/trap.c:139 0x806a9ea

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: __kernel_rt_sigreturn+0x0  ()

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: abort+0x17e  (/lib/i386-linux-gnu/libc.so.6)

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: __assert_fail+0xf8  (/lib/i386-linux-gnu/libc.so.6)

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: /usr/src/tvheadend/src/epggrab.c:660 0x8056487

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: /usr/src/tvheadend/src/main.c:394 0x804b89a

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: __libc_start_main+0xe7  (/lib/i386-linux-gnu/libc.so.6)

Jun 16 19:04:29 xbmc-mmb tvheadend[2234]: CRASH: ??:0 0x804ae51

Thank you for your work! TVHeadend works well, but the "wrong" EPG is very annoying! Big Grin

**adamsutton** · 2012-06-16 21:22

Oops,

did you spot the obvious error Wink

I left an assert(0) in the code that I put in while testing the configuration migration, doh!

On a positive note: that gave me a chance to warn you to BACKUP your configuration as my code alters things. However in light of this I've slightly tweaked my code so that it will actually keep all the old files and use new locations for now. So you should in theory be able to run the old and new without them interfering with each other.

Note: because of the assert() your config will be untouched Smile

If you are going to give the code a try, best bet is to pop along to #hts and have a chat. It's still in flux at the moment and although I'd love to have people try it out I definitely can't give any garauntees about performance at the moment.

P.S.
I've fixed the "bug" and it should work now (fingers crossed).

andyb2000 · 2012-06-17 19:57

Hi Adam,
Sounding good. I was about to upload the stuff I'd done so you could take a look but I think you've beaten me to it! I've been tweaking with the tv_grab_dvb_plus-0.2.1 code, basically what was mentioned here and in the tvh forums, so you had to swap the channel over to the epg background audio channel and do the grab then. I modified the source of the main.cpp so that the application defaulted to the sky epg grabbing, and also forced the devices it was talking to, as defaults didn't work right for me, I also modified it to cut out the commented data that it added to the xml output as that caused xmltv import errors.

However, I thought it was all working...Until today, and found that bits of my EPG are now missing, and it appears the grabbing isn't working right, so I'm going to grab from your github and see if we can progress on that and then submit it up to Andreas.

I'll try and catch you in #hts but I'm around just inbetween family, etc, so tricky!

**adamsutton** · 2012-06-17 22:09

Andy,

what's your IRC nick? just so I can keep an eye out for you?

Yeah same thing here, trying to find as much time as I can but difficult with family etc...

My branch is very different from andoma's I've been speaking to him throughout and I'm hopeful he will accept the changes but it is a complete re-write of the EPG and grabber code (and some other stuff is likely to have to change as a result of this latest stuff).

Also my github branch does NOT contain any of the opentv stuff yet, I've only got that in a local branch as I want to keep it separate from my main work and I'd planned to make it available once I had something that actually worked. But if your keen send me a message and I'll see about putting the branch up on github.

As for the opentv stuff I'll summarise where I am now (slightly moved on from last update):

1. Basic configuration structure is in place, but so far only skyuk config exists and its forced ON. How to actually handle the config is a TBD, but not a particularly tricky task.

2. Most of the integration into TVH with respect to setting up to the receive the data exists, so as long as you tune to the right channel it'll receive the data.

3. Most of the parsing of the OpenTV data is done, however its currently just dumped to stdout.

Things I need to do:

1. Sort out the background tuning.. I'll probably leave this to the end.

2. Modify the TVH DVB tables code, currently only one filter can be added per PID. This is a problem since the standard EIT code installs on 0x11, but opentv code also needs this for some other tables that give it channel info. Shouldn't be too tricky.

3. Process the data into EPG data structures and stick into the EPG.

Given no other obstacles I'd say its a couple of days to get it to a working state. But more likely it'll take a couple of weeks to find enough time.

Adam

adamsutton Moderator Posts: 239 Joined: Jan 2012 Reputation: 14	2012-06-14 21:37 Post: #11 Drae, thanks for that I'll have to look into it that. I was aware that a 7 day EIT EPG existed on some services. I wasn't aware that the series link information was available in the stream. But that would be very useful I'll have a look at that VDR/MediaPortal plugins when I get a chance. Adam
	find quote

the_mouse Junior Member Posts: 38 Joined: Jun 2012 Reputation: 0	2012-06-16 01:44 Post: #15 I am using Tvheadend the main issue I have is with the EPG. Here in New Zealand the guide is transmitted using MHEG5 over DVB-T and covers 8 days. I dont know where else in the world is using this format but no DVR software seems to support it. Is there a possiblity this could be looked at?
	find quote