EPG (TV-Guide) data-store code development discussion for the unified PVR front-end

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Information  EPG (TV-Guide) data-store code development discussion for the unified PVR front-end
Post: #1
Hopefully this will serve a place to continue this discussion in the User Area concerning the methods for storing, accessing and displaying the EPG data within the XBMC application. (Note that I'm not talking about the specific mechanisms that the individual plugins will use to communicate with the PVR server. There will need to be threads set up for those separately, when the time comes).

This is partly to encourage the open discussion of these issues in the development of a XBMC-class EPG, but also because I need this sort of information for my proposal! Wink
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Lightbulb   
Post: #2
I'll start the ball rolling with thoughts on the EPG data-store.

So a major issue to iron out is how the EPG data is stored in XBMC before being displayed by the GUI, and how the data will be populated by the 'client' role of the XBMC app. This will have to be a unified solution for the multitude of different PVR applications out there.

There has been a suggestion that XML is used, for a number of reasons:
  • It's likely that XML will be one of the methods for Server->Client communication of EPG data
  • Some people will want to use XMLTV 'grabbers' to populate their EPG
  • Python scritps could easily output an XML file
  • The previous work done on the XMLTV GUI could be used as a base to work from
I can certainly see XML being used by the client part when that is how the server transfers the EPG, but I don't think it's suitable as the internal data store.

Perhaps using the SQL db which is already being used for Music and Video libraries would be more sensible. The 'client' role would populate the EPG related tables and the GUI would just query the db for the data.

Someone with more knowledge of the performance benefits of this might want to add their views here... It's quite possible the EPG would be updated every minute, particularly if the user was taking advantage of EIT for changes in program start times etc. With the possibility for very large EPGs coming off of digital satellite, I imagine SQL querying to be far more efficient than XML parsing (on the fly).

Plus the Video library code could be easily modified to include the TV recordings as a root folder.

what does anyone think?
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,181
Joined: Oct 2003
Reputation: 176
Post: #3
I guess it depends on:

1. Amount of data - can we get some info on typical sizes?
2. Frequency of data updates (i.e. how often do we need to add to the datastore?).
3. Frequency of reading the EPG from within XBMC (i.e. how often do we need to check the datastore for refresh, or should this be pushed from 2?)

With that info, it should be easy to setup some test cases to see whether XML parsing will suffice in terms of speed and memory usage and so on, or whether we need to sql-ize it.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Post: #4
I'll try and come up with some values for typical data sizes before I go to sleep tonight. If we imagine an average 14 day data set with ~200 channels it's probably not a massive amount but I'm thinking more along the lines of polling the data within xbmc which will require a more sophisticated storage system.

Plus it seems conducive with getting recordings into Videos section..

In respect to the frequency of updates, if the EPG is to be truly 'live' then I imagine frequent transactions from the client role - even as much as once a minute?
find quote
Nick8888 Offline
Fan
Posts: 716
Joined: Jan 2007
Reputation: 0
Post: #5
1.6mb here from shepherd epg australia.
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Post: #6
hi nick,

could you let us know the number of channels & days ?
find quote
Nick8888 Offline
Fan
Posts: 716
Joined: Jan 2007
Reputation: 0
Post: #7
oh thats only 10 or so channels as well. its large because it has all the descriptions, actors etc
find quote
Nick8888 Offline
Fan
Posts: 716
Joined: Jan 2007
Reputation: 0
Post: #8
sorry I posted before your reply. 7 days. Ill send you the xml if you want. tried to put it on pastebin but it was too big.
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Post: #9
well I'm starting to get tired now but I have some results

UK RadioTimes: 220 channels and 14 days grab = 46MB XML file with the following schema:

[HTML] <programme start="20080408090000 +0100" stop="20080408093000 +0100" channel="digital.scotland.bbc2.bbc.co.uk">
<title>The Story of Tracy Beaker</title>
<desc lang="en">Drama series about a young girl who is looking for a foster home.</desc>
<credits>
<actor>Dani Harmer</actor>
<actor>Montanna Thompson</actor>
<actor>Chelsie Padley</actor>
<actor>Joe Starrs</actor>
<actor>Sonny Muslim</actor>
<actor>Jay Haher</actor>
</credits>
<category lang="en">Children</category>
<video>
<aspect>16:9</aspect>
</video>
<subtitles type="teletext" />
</programme>
[/HTML]


Having that many channels seems like madness to me (who watches very little television) but the 14 days must be a realistic average, as people will want to plan their recordings for holidays away etc.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,181
Joined: Oct 2003
Reputation: 176
Post: #10
Right. Parsing a 46MB file is completely unrealistic on xbox, so an alternative storage system will certainly be required.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 9
Location: Sweden
Lightbulb   
Post: #11
XMLTV formated listings.xml for input from the client and a own epg.db SQL database for the storage does sounds logical.
However I do not think it should be the clients job to populate the SQL database directly, instead the clients job should be to provide the XMLTV formated listings.xml file to the XBMC EPG API, then it will the the job of XBMC EPG API to parse that file and populate the SQL database (which in turn the EPG GUI part will use).

Maybe look at how MediaPortal handle/code their EPG? MediaPortal is coded in C# but just to get an idea (or port parts)

Confused

By the way, here is some more reading matertial on the subject => http://forum.xbmc.org/showthread.php?tid=12948


PS! Remember that those bloody Americans have like 1000-channels each Shocked but then usually only a 7-day EPG Eek
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Post: #12
Gamester17 Wrote:XMLTV formated listings.xml for input from the client and a own epg.db SQL database for the storage does sounds logical.
However I do not think it should be the clients job to populate the SQL database directly, instead the clients job should be to provide the XMLTV formated listings.xml file to the XBMC EPG API, then it will the the job of XBMC EPG API to parse that file and populate the SQL database (which in turn the EPG GUI part will use).

another long post, apologies...

The issue I see with that approach, is that all client's will have to translate the EPG from whatever protocol they receive it from the server into one big XML file before the EPG updates the db. For instance people using VDR and EIT, will grab the EPG via SVDRP, then will have it converted to XML, then inserted in the db. Remember a lot of people won't be using XMLTV, and most will want to see the server's version of the EPG for consistency. If the server want's to grab XMLTV listing and blend it with program changes via EIT, then that's all the better for us.

Of course with this approach changes to the DB structure may need to be replicated by all the clients, however that problem could be solved by a thorough design for the DB in the first place.

How are live updates to the EPG going to be managed, with XBMC parsing a BIG file each time a program start time change happens? I'm also wondering how status messages for programmes, such as "Recording", "Schedule Clash" etc would be reported to the GUI. Flicking bools in the db could be an efficient way to manage these changes.

If in the future, the various client <-> server communications were improved, so that updates were pushed from the server, having the client update the db directly with a couple of SQL transactions seems more sensible than pushing an EPG change to the GUI, which then parses the entire XML file and looks for changes accordingly. This might indirectly cause slowdowns in the GUI.

I'll try and have a look at those sources some time today or tomorrow. Seeing how other people do this is definitely a top priority for me :-)
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Post: #13
re-reading your post, I think we're talking about the same thing..!!

The XBMC EPG API in tandem with the client would populate the EPG.

I was imagining this EPG API would just offer some friendly ways to access the DB, rather than performing file parsing roles itself.
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 9
Location: Sweden
Question   
Post: #14
alcoheca Wrote:How are live updates to the EPG going to be managed, with XBMC parsing a BIG file each time a program start time change happens?
Why not aim to only input a full (one or two week) EPG data-file at the initial-setup time, then try to do incremental daily updates of the EPG data?, that way database would always contain one week (or two weeks) listings, ...instead of it updating the full EPG once a week (or once every other week) which would mean the database would not always have the full one week / two weeks of EPG data to display.

alcoheca Wrote:If in the future, the various client <-> server communications were improved, so that updates were pushed from the server, having the client update the db directly with a couple of SQL transactions seems more sensible than pushing an EPG change to the GUI, which then parses the entire XML file and looks for changes accordingly.
Yes, that makes sense.


PS! Not really important nor relative just yet but should "scheduled recordings" be stored in the same SQL database or a separate?
find quote
alcoheca Offline
Team-XBMC Developer
Posts: 265
Joined: Feb 2008
Reputation: 6
Post: #15
Gamester17 Wrote:Why not aim to only input a full (one or two week) EPG data-file at the initial-setup time, then try to do incremental daily updates of the EPG data?, that way database would always contain one week (or two weeks) listings, ...instead of it updating the full EPG once a week (or once every other week) which would mean the database would not always have the full one week / two weeks of EPG data to display.


This may suit some people, but digital TV has advantages over printed schedules - program changes can be replicated via EIT to the users. With XMLTV you don't have this option, the listings providers won't want hourly grabs, and you can't just grab the updates, like a rsync session. It would make sense to have a twice weekly, or weekly full update on the SERVER which could run when it was idle.

If you incorporate EIT into the mix (at the server side), then the facility for hourly, half hourly incremental updates would be a great addition. Client updating epg.db via EPG API.

I think if the onus on EPG handling is left with the server, then we can ensure we have consistent data across the board, and not be replicating tasks in a multi xbmc frontend setup.

Perhaps this is straying from the topic - if the client deals with all of this then GUI EPG doesn't have to care and just pulls the schedules from the db. Big Grin
(This post was last modified: 2008-03-27 17:00 by alcoheca.)
find quote
Post Reply