Developing MeD's Movie Manager interface to XBMC - assistance is needed
#1
Hi,

I'd like to start developing an interface between MeD's Movie Manager (pulls movie info from IMDB and stores it in a MySQL database: http://xmm.sourceforge.net/index.php?menu=main) and the XBMC. Currently my main concern with XBMC is that the movie database is stored on the XBox. I would like to have a central database using MeD's that a number of XBox's could query. Each entry would be able to return the file path, thereby allowing you play movies, but more importantly, it would allow to search through your movies for what you want, search by actor, genre etc. whatever you like. You would also have the option of viewing other information about the movie. Sifting through a file browser for the movie you want has now become a very large task...

I was going to start with a Java Servlet with which XBMC could communicate. But I need to know where to start. What is the best way for XBMC to perform queries? Would a list of movies returned in XML format be appropriate? I'm not really concerned about the XBMC end at the moment, I'll focus primarily on servlet side of things to start. I'll be able to get a start on it and leave figuring out XBMC development till a later date.

I'm a Java programmer with experience with servlets primarily. I have worked with C++ in the past.

The beauty of this method is that MeD's is also Java based, thereby allowing it to work with any future ports of XBMC!

Any help would be great, thanks!
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#2
Hi there, and welcome.

First off, one presumes you've actually tried XBMC's library features out. We use sqlite to do the db side of things, which in it's current form cannot connect to a non-local database (to my knowledge - if you know better, please let me know!). Converting sqlite stuff so that it runs as a dll will open up the possibility of having a non-local database, so that's one way you may want to look at.

Otherwise, something like XML would be a suitable method of passing stuff across. With the nice new virtual python "plugin" folder system we have under way, this would allow you to integrate it with XBMC without having to code any C++ at all (i.e. all done in python) and I would suggest that is the way you go, as I'd think that we'd be not all that keen on having it in SVN for a feature that will be useful for only a small amount of users.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#3
Excellent! That's exactly the answer I was looking for! I'll be back once I have a suitable servlet ready.

I think using XML would be the best idea as it would allow communication with other software not just XBMC. I'd like the info server to be as portable as possible.

Almost all the people I know who use XBMC view there files using a network share on a PC. This would pretty much allow anyone to have an info server for there media. Could easily be expanded to include TV shows also.

Guess I'll be learning some Python in the near future!

Thanks for the quick reply.
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#4
wouldnt it be usefull if we could load multiple databases then ?
like when someone connects to several computers, and all those computers have their own database, xbmc all loading them into one.
Reply
#5
Your already seeing where I'm going with this!!!

This is why I think XML would be best. Once the schema of the XML is formalised, and generified, you would be able to easily combine the queries. Then, no matter where the data comes from, whether it already is in XML, from a database or even text files you would have everything. Plus it would be database independent.

It will also let each media location be responsible for maintaining it's own media and the information about it. A simple update could be run on all media locations, updating the database in XBMC. Then all searches could be performed locally.

In fact it would be quite easy to write a generic servlet to access data of any type by using a data persistence layer. By simply throwing Tomcat onto your media PC and writing a config file, XBMC would be able to comunicate with it.

There's potential there for this to be very cool!
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#6
phunkyfish Wrote:my main concern with XBMC is that the movie database is stored on the XBox. I would like to have a central database using MeD's that a number of XBox's could query
why not write an python-script for the XBMC client-side and a interface/parser on the MeD server-side to sync the two databases on a regular bases (or on demand). So instead of having XBMC querying the MeD database per movie you just syncronuze the whole database and then just let the end-users use XBMC own query functions to query the local database.
http://xboxmediacenter.com/wiki/index.ph...on_Scripts

Huh

phunkyfish Wrote:Almost all the people I know who use XBMC view there files using a network share on a PC. This would pretty much allow anyone to have an info server for there media. Could easily be expanded to include TV shows also.
I do not see how (or why) you would run this via file-mode in XBMC, you would have to use library mode.

Like Jonathan said/asked, have you tried XBMC in library mode?, it already does all you want, only locally, see:
http://xboxmediacenter.com/wiki/index.ph...C_Database
http://xboxmediacenter.com/wiki/index.ph...eo_Library
http://xboxmediacenter.com/wiki/index.ph...e_Scrapers
http://xboxmediacenter.com/wiki/index.php?title=IMDB
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#7
Gamester17 Wrote:why not write an python-script for the XBMC client-side and a interface/parser on the MeD server-side to sync the two databases on a regular bases (or on demand). So instead of having XBMC querying the MeD database per movie you just syncronuze the whole database and then just let the end-users use XBMC own query functions to query the local database.
http://xboxmediacenter.com/wiki/index.ph...on_Scripts

Yes, I think this is exactly what I want. And as below you could sync with MeD on multiple PC's if you so wished. I would only be using it on a single file server, but it should allow you to be able to specify multiple MeD/Other info server database locations...

Gamester17 Wrote:I do not see how (or why) you would run this via file-mode in XBMC, you would have to use library mode.

Like Jonathan said/asked, have you tried XBMC in library mode?, it already does all you want, only locally

Sorry, I did not explain myself fully. I use the file manager currently because the scrapers don't work very well. I have used the library in the past but it is somewhat lacking. The main problem with XBMC is getting the correct data into the library. Once the correct data is in there the Library functionality is almost perfect. Let me explain:

1) Movies - When you scan a bunch of movies into the library a fair portion of them will be scanned incorrectly, i.e. the info for the wrong movie will end up in XBMC, as an example, Apocalypto will come up as Apocalypse Now. Some scrapes are far worse, with no resemblance to the correct movie at all. It's about 50-50 over a hundred movies or so, half right, half completely wrong.
Also there is no way of selecting multiple genres, how would you show all "action comedies" or show all that are action or comedy?

2) TV Show - I have my shows organised in the following directory structure: "TVShowName->SeasonNumber->Files", each file is then labelled "1x11-showName-otherText.avi". If I try to scan in one directory (i.e. one show) all I get is a pile of folders labelled as numbers and nothing else. Selecting the "folder contains one TV show" option makes no difference. I guess the show name should be taken from the initial folder name and season and episode from the avi file. I could rename all the TVShow avi's so they can be processed by regular expressions... But that's a rather large job... Anyway, not really bothered by TV shows. My directory structure works well enough.

I have a few ideas on how to sort the problems with the movies. One is to have a "video.xml" in each directory stating the "movie name", "year", "imdb ID" and "video files", could have more information if you wish. This would make the scraping task much easier as you know exactly which movie it is. The "video.xml" could have details on any scrape site, and could be extended for use with TVShows also. It would make directory structure irrelevant. It's quite easy to add a video.xml to each TVShow/Movie as it's added. But creating them for every Movie/TVShow when you have a lot of them could be a huge task. However, it would yield better results than the current scrape system. Also, if directories are moved it would be easy to relink the file location to the database. Bear in mind this is all without ever connecting to an external info server. If you think this idea would be worthwhile, I'm already writing a schema to incorporate all video files, any input? Note that this is only for simple Movie/TVShow info. It would just enable better scraping.

The reason for the info server is to be able to have a central location for the information. If your XBox dies, the info is gone. Plus if you've watched a movie on one XBox and then on another XBox in the house it's marked as unwatched. If this information is stored centrally it makes more sense. There would even be option of adding multiple "watched/unwatched" users to the info server. So even if you watch stuff on different XBox's it will still show what you havn't watched correctly no matter who you are.

Let me know what you think? I'm going to use the "video.xml" idea with MeD's, if we formalise the schema between XBMC and MeD's all the better, you would have the option of supporting it in the future. You could even have an "only scan video.xml" option?
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#8
This is ridiculous. I'm trying to post a reply, if I enter the reply I want it will never display, even if I remove the quotes. Anyone encountered this problem before?
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#9
Have approved your post - not sure why it's being moderated. Note that if it happens again, there's no need to keep replying - it will be approved as soon as it's noticed. Will ask the admins why it's being moderated and see if we can sort it out.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#10
Scratch that. It appeared after an hour? I hope it doesn't come up multiple times now. Any moderators around feel free to tidy up any erroneous posts. I hate wasting page real estate!Laugh
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#11
Some answers:

1. XBMC's database is SQL - thus, any query you like can be constructed as long as the code is added to construct and execute said query. I suggest extending the music Smart Playlists is the way to go in this regard - it allows basically anything.

2. XBMC will support that filesystem layout for scanning TV shows as long as you set up the regexp so that it'll work. You want to assign TV shows to the folder that contains the <tvshowname> folder, or assign "single TV show" on the <tvshowname> folder. The default regexps will then work I suspect with your show filenames. You are just setting content on one folder too far in, so it looks at "1" as the show name.

3. XBMC already supports ".nfo" files which can be XML with all the stuff that IMDb and other sites can return - see the online manual for more, and you can export the db as XML.

4. As far as XBMC goes, I would personally like to see how SQLite responds over the network - having XBMC's own native db file available on the network, assuming the retrieval isn't too slow would be the way to go IMO for multiple installs. As I said, this can be done in one way by building sqlite as a dll, thus the filesystem stuff will be transparent, and sqlite will see it just as a local file no matter where it is. I'm not sure if this can be done in a more efficient way or not, as I'm not a db guy.

Hope this answers some of your questions - I'm interested to see where you see this going - many have requested such a feature in the past, but noone has yet come through with a solution that works gracefully.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#12
good luck with this, i am anxious to see how it turns out and will be watching it closely. Smile
Reply
#13
Cool, it seems you guys have already done a lot of the work for me!

Firstly SQLite:
-------------

We know that SQLite can't access a remote database. But is the converse also true? Can a remote machine, let's say a PC, access an XBox's local database ? I mean without copying it over, editing and copying back?

Concerning TV Shows:
---------------------

I'd say I just need to create an "advancedsettings.xml" to add a new regular expression for filenames like "1x05-showname.avi". There doesn't seem to be an existing expression for filename starting with the season number they all appear to want some text first. Quick question, are the regular expressions in "advancedsettings.xml" processed before or after the default ones? I'm presuming before... I'll get back to you on this if I don't have any luck.
Since your only searching for Season and Episode numbers why by default expect a "." or a "-" first?

The following expression:

<regexp>[\._ \-]([0-9]+)x([0-9]+)[^\\/]*</regexp> % foo.1x09*

could instead be:

<regexp>[\._ \-]?([0-9]+)x([0-9]+)[^\\/]*</regexp> % foo.1x09*

matching 0 or 1 dots or dashes removing the need to add an "advancedsettings.xml"?

And moving on:
---------------

As far as development goes I think I'll stay away from SQLite as a DLL. It would be the ideal solution in a way but it's been a while since I've used C++ and was always a Borland guy at the time! We didn't like DLL's very muchLaugh

I'm going to try converting the information required to XML, eventually coming up with a generic format for library information. I'll use the NFO method for scraping, so that it can work automatically and correctly. I'll use the same NFO method that XBMC uses with MeD's. Then I will be able to use Python (used it in it's infancy a long time ago!) to add the XML info to the library. I'll make this work both ways, so that both MeD's and XBMC can be converted to the same XML format so that both XBMC->MeD's and MeD's->XBMC are possible. At his point combining multiple XML files into one if there are multiple media locations should be easy.

Finally I'll try expanding the search functionality in the library to allow more complex searches. Not sure if I'll use python or delve into C++ for this. Advanced searching may be something that everyone might want so C++ might be the way to go on this. Either that or I'll bug some of you guys into implementing it!!! But this will definitely be the last thing done.

Anymore more feedback/direction is very welcome...
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply
#14
1. Regarding SQLite: If it's built as a dll it'll have no problem accessing a db over the network. This is IMO the first thing to try as far as performance goes. I'm not sure how well it'll work as a DLL but that would be the way I suggest would be "easiest" to do. Alternatively (probably easier), you'll need to build it so that it uses the XBMC filesystem (CFile) instead of whatever it normally uses (probably just standard I/O) - this will mean it includes stuff from XBMC, so will require it be included as a separate project within the solution, but that's no problem.

Getting something to connect to the local (on xbox) SQL db is not really doable, unless you want to completely replace it (i.e. just copy a new version over, or run an Import from XML).

2. Regarding TV show regexps - obviously we're open to altering the defaults if there's a reasonable need to do so - will chat with spiff next time he's around.

3. Via python or C++ is fine for this - the Music Database exposes a general SQL query interface (possibly that only returns fileitems - haven't really looked at it), and I expect the Videodatabase either already does or can be made to do so. Combining this with what we have in the python plugin folder stuff will mean you can do lots of fun stuff.

Cheers,
Jonathan
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#15
With the advent of XBMC doing all scraping correctly maybe I could to do this with just a servlet and a python script. The servlet could use the XML file as it's database it would require a little more intelligence on the servlet end, but it's possible. At a latter stage a database backend could be added to the servlet. Or even use SQLite as it's database. Come to think of it, the servlet could just use the XBMC database. When the point came that XBMC's SQLite DLL could access remote databases integration would be flawless. No point in creating a new database layout if I don't need to. At this point the servlet could just be used for remote access of the database so you could view your library when off the local LAN.

Yup, just checked, there's a Java wrapper for SQLite. Heehee, just read the post again, you can see my thought process change as I wrote it!!!

Laugh
Maintainer of Enigma2 PVR addon: repo, docschangelog
How to create a full debug: here
Reply

Logout Mark Read Team Forum Stats Members Help
Developing MeD's Movie Manager interface to XBMC - assistance is needed1