[jpichie]

I heard on one of the video tutorials for upnp sharing in 13.2/14 that it is still being worked on for a better server/client setup.
I am wondering if there is any progress on this (upnp data source showing up in tvshows and movies on main menu bar, instead of under video files). Also, will there be a server service at some point?

I know mediaportal has this working with MP2 alpha right now, and I am on the fence as to which to use as my main at home MP or XBMC.
I could say as a long time MP1 user, and MP2 tester, that XBMC is really only missing a nice server client to get my vote. I am very impressed with the speed, as well as the OS variety it supports.
thanks for any answers!

[Montellese]

I've been working on it (not so much in the last 2-3 months due to very limited time) and there's a lot of stuff already working but there's also a lot of stuff not working and some really difficult obstacles in the way.

The basic idea is that if XBMC/Kodi detects another XBMC/Kodi with UPnP library sharing enabled it will prompt you with a choice to import items from that other library or to ignore it. When choosing to import you can choose which media types (movies, tvshows and/or musicvideos for now) and then it will import all the items from the other library with all the metadata available. You can then manually trigger synchronising the items from the remote library or it happens whenever the remote library is detected. You also get to choose whether you want to synchronise at all (maybe you want a one-time import) and whether to synchronise watched, resume point and last played information with the remote library.

But there are a lot of things that aren't working yet like the general GUI integration, handling cases where the remote library e.g. offers the same tvshow as you have in your local library, removing all items from a remote library and a lot of other stuff.

My ultimate goal is to also rewrite the current source handling to work in the same way as the remote library importing. That way we could use the same system/logic to import media items from many different sources (filesystem, upnp, stuff like MediaBrowser integration etc.).

Maybe there'll be another video from the next DevCon where I can demo the current state of my work.

[Fice]

nice to hear someone is actively working on this.
a few weeks ago I decided to start working on something similar myself. I don't know if you remember my 2013 GSOC proposal, but the initial idea was to have a Database that stores changesets of the metadata tables. Those changesets could then be exchanged.

I have a working Proof of Concept of such a Database "framework". It basically has triggers that automatically store the changes in a spereate table.

You can then manually trigger synchronising the items from the remote library

I have registered a commit hook with sqlite, that gets called whenever something has changed. (that could be used to trigger the sync automatically).
I don't know if mysql has some similar function, but worst case would be to poll for changes in a background task.
And since we can easily query the db (well the query right now has lots of joins ) for new changesets, we do not need to sync all items in our DB.
Those changesets also have a Timestamp, so if the value changed on both devices, we could use the newest value.
The changesets also have the old and the new value. That could be useful to apply deltas (on some fields). Let's say you have a new movie and you watched it once on your tablet and once on your HTPC but both devices haven't synced yet. If we sync now and just override the values by the values the partner gives us, we will end up with a play count of 1 on both devices. By using deltas, we could figure out that the user has actually watched it twice. (Don't know if kodi actually stores a play count for movies, but for music it does).
The framework is really easy to use: You just need to implement Two functions in the CDatabase derivates (one function that returns true and one that returns an array of tables that should have changesets). I have already done that for music and video dbs but it would be no big deal to add that functionality to future dbs.
TODO: Rewrite the triggers and get it to work with MySQL (sqlite only for now).

On top of that, I started to implement the 'UPnP Content Sync Service' which is ment for sharing and updating files (optional) and their metadata (that's what I'm trying to do). I have to admint that the specs are a little sketchy on some key points and it disallows the syncing of the same item with more than two devices. So I was going to loosely follow that spec and make some alterations along the way, but we might be able to keep interoperability with non Kodi devices that implement that spec (To be fair, I don't know any that do, so that has rather low priority). I'm still fighting with libUPnP (great documentation they have) but I'm getting there.
TODO: well... nothing to show for yet, so quite a lot of work to be done.

@Montellese: Might you be interested in joining forces on that one? I've looked at your media_import branch (only in the network/upnp folder) and it seems you implemented some custom behaviour for the syncing. Are you interested in using my work on the Sync Service UPnP Specs?
My git history on my branch is nowhere near usefull (I fucked up two squashes and then decided to just don't care about a clean commit history anymore), but if you want I could atleast clean up the DB changes and provide them if you want.

EDIT: just cleaned up my branch a bit and pushed it: syncService

[jpichie]

Montellese, Fice - this is my only knock to open source software, yes anyone could edit and create things, but it is nice to have some things standardized/worked on by a team.

Montellese, it is nice to hear this is being worked on. This may just be my opinion, but maybe getting the uPNP shares to show up correctly on the main menu would be a good working 1st step. Not sure if this is on your end, or on the skinning end.

Auto attaching upnp sources would be a nice thing in the future.

I guess for now, I could try the mysql database, or just attaching network shares until a better/ more complete upnp/dlna solution is available
thanks

[Montellese]

nice to hear someone is actively working on this.
a few weeks ago I decided to start working on something similar myself. I don't know if you remember my 2013 GSOC proposal, but the initial idea was to have a Database that stores changesets of the metadata tables. Those changesets could then be exchanged.

I think I vaguely remember.

On top of that, I started to implement the 'UPnP Content Sync Service' which is ment for sharing and updating files (optional) and their metadata (that's what I'm trying to do). I have to admint that the specs are a little sketchy on some key points and it disallows the syncing of the same item with more than two devices. So I was going to loosely follow that spec and make some alterations along the way, but we might be able to keep interoperability with non Kodi devices that implement that spec (To be fair, I don't know any that do, so that has rather low priority). I'm still fighting with libUPnP (great documentation they have) but I'm getting there.
TODO: well... nothing to show for yet, so quite a lot of work to be done.

TBH I didn't even know this UPnP specification existed. I only ever looked at the UPnP AV specifications. Thanks for the hint. I've taken a peak at the specs and it looks like it is only really possible to implement this in XBMC with the changeset tracking you are implementing as well. Without that it would probably be impossible.

Yeah Platinum SDK is a bit of a mess and I'd really like to replace it with something better but there's not much available and it's a hell of a lot of work.

@Montellese: Might you be interested in joining forces on that one? I've looked at your media_import branch (only in the network/upnp folder) and it seems you implemented some custom behaviour for the syncing. Are you interested in using my work on the Sync Service UPnP Specs?
My git history on my branch is nowhere near usefull (I fucked up two squashes and then decided to just don't care about a clean commit history anymore), but if you want I could atleast clean up the DB changes and provide them if you want.

The changes to UPnP are actually the smallest part of my media_import work. All I did there was implement the Search method with support for searching for specific upnp:class attributes. Furthermore I've extended support for certain metadata (some of that work has already made it into master and will be part of Helix) and introduced new upnp item classes for tvshows because that's not covered by the UPnP specification. A more generic approach for this would be welcome but extending the upnp item classes is not against the specification.

When I first started out (based on the work pieh did in his GSoC 2013 project) I completely focused on the UPnP stuff but I have changed my view since than and what I'm trying to achieve now is a generic framework for "importing" items into the library (from anywhere). It consists of an import manager, media importers (which could be binary addons at some point) and abstracted logic to add specific media types to the library. The idea is that the user specifies a location from where items should be imported. Then there's a task that is responsible for retrieving the items, a task to synchronise the imported items with the items already in the database (which will result in a list of changes that need to be made to the current library, a potential task to get additional metadata for the imported items (scraping) and a task to apply the changes to the library. That way the source could be a local/remote filesystem, a upnp contentdirectory provider, a plugin or something else and all it would need would be a source-specific importer and the rest would be handled the same.
This concept is currently missing the synchronisation of states like watched and lastplayed back to the source though.

Montellese, it is nice to hear this is being worked on. This may just be my opinion, but maybe getting the uPNP shares to show up correctly on the main menu would be a good working 1st step. Not sure if this is on your end, or on the skinning end.

What do you mean by "show up correctly on the main menu"? You can add a custom main menu entry that simply links to the UPnP share in which case you would always browse the UPnP share directly. It's not really an integration into your local library though (which is what I'm working on).

[RockerC]

Was all the work described in these two other threads by alcoheca ever merged into mainline?
http://forum.xbmc.org/showthread.php?tid=127949
http://forum.xbmc.org/showthread.php?tid=162225

nice to hear someone is actively working on this.
a few weeks ago I decided to start working on something similar myself. I don't know if you remember my 2013 GSOC proposal, but the initial idea was to have a Database that stores changesets of the metadata tables. Those changesets could then be exchanged.

I think I vaguely remember.

Referring to this one http://forum.xbmc.org/showthread.php?tid=160963 right?

When I first started out (based on the work pieh did in his GSoC 2013 project)

And that is this one http://forum.xbmc.org/showthread.php?tid=165024 or?

[Montellese]

Was all the work described in these two other threads by alcoheca ever merged into mainline?
http://forum.xbmc.org/showthread.php?tid=127949

Yes, at least that which he achieved during that GSoC project.

http://forum.xbmc.org/showthread.php?tid=162225

No, since he didn't find the time to work on this project.

When I first started out (based on the work pieh did in his GSoC 2013 project)

And that is this one http://forum.xbmc.org/showthread.php?tid=165024 or?

Yes but from the original code there is very little left since it was very specific to the UPnP approach.

[jpichie]

Ideally what I am hoping for, would be kind of what you mentioned in your 1st post, but more simple.
Find uPnP shares on the network, and connect to ONE as a server. Use it's library as your own, updating your "local library" but linking to the server.
This is something that MP2 has working nicely, and would be fantastic if it was working with XBMC in the future.
thanks

[Montellese]

Ideally what I am hoping for, would be kind of what you mentioned in your 1st post, but more simple.
Find uPnP shares on the network, and connect to ONE as a server. Use it's library as your own, updating your "local library" but linking to the server.
This is something that MP2 has working nicely, and would be fantastic if it was working with XBMC in the future.
thanks

That's what I'm working on but I'm simply choosing a more generic approach to not limit the logic to a single UPnP server. You can always use it the way you want by only importing items from that ONE server. And I want to be able to have a local library on the side so in the end you have a library mixed with items from different sources/servers.

Agreed that your simplified approach would be a lot easier to achieve with regard to the added functionality but it would require a lot of extra work to stop people from adding local sources etc as well.

[jpichie]

true enough, I look forward to testing any progress made.
let me know if you need anything tested.

[Fice]

nice to hear someone is actively working on this.
a few weeks ago I decided to start working on something similar myself. I don't know if you remember my 2013 GSOC proposal, but the initial idea was to have a Database that stores changesets of the metadata tables. Those changesets could then be exchanged.

I think I vaguely remember.

Referring to this one http://forum.xbmc.org/showthread.php?tid=160963 right?

Sort of. My initial proposal was more like generating changesets on every device and having a way to exchange them. (Idea was that you could make a file locally available on yur tablet, watch it on the train or wherever, and once you're back home, all other devices would know that you have watched them). It then moved in the direction to implement the possibility to take that file with you and actually play it (And the metadata syncing was dropped). I edited the first post to reflect the final proposal, not the initial one.

TBH I didn't even know this UPnP specification existed.

I only found it by accident as well To be honest, it also doesn't seem like the spec really took off. If anyone knows of some software/device that actually implements that spec, I would love to hear about that.

Then there's a task that is responsible for retrieving the items, a task to synchronise the imported items with the items already in the database (which will result in a list of changes that need to be made to the current library, a potential task to get additional metadata for the imported items (scraping) and a task to apply the changes to the library.

Can the media importer overwrite the task that is responsible to generate the changesets? In that case I could implement a 'UPnP Content Sync media importer' (which woul only differ very sligthy for the initial importing from normal UPnP servers), and I could create a specialised task that listens for changes.
It would probably be more robust to get the changesets from the source itself (onyl if possible of course) than trying to create them yourself. Image you play ItemA an DeviceA, then you play the same item on DeviceB and then they sync: Play count would be the same on both devices so the resulting play count would be one, when in fact the user has watched the item twice. Having changesets that know the playcount has been increased by one, would solve that issue (and a few more).

Code:

virtual bool CanUpdateMetadataOnSource(const std::string &path) const { return false; }
virtual bool CanUpdatePlaycountOnSource(const std::string &path) const;
virtual bool CanUpdateLastPlayedOnSource(const std::string &path) const;
virtual bool CanUpdateResumePositionOnSource(const std::string &path) const;

From looking at your code, you're only planing to push changes to the source, when the 'play statistics' have changed, right?. Would it make more sense to just give the IMediaImporter implementation the whole changeset data (seems to be a 'Set<Field>'?) and the IMediaImporter can decide what he can push back to the source... An addon might not be able to push any changes or it might only be possible to send the playcount, but for the sync service we can push pretty much everything. That way, if I add a tag to one of my items, it would instantly be pushed to all other kodi Instances that are online.

For now I will just focus on implementing the SyncService, since most of the changes are in libUPnP anyway. Once I have sth. actually working I will see if I can implement a UPnPContentSyncMediaImporter.

FYI: One issue I thought of was the following scenario: If you start to watch a movie in the living room, then you stop in the middle, turn it off, go to your bedroom turn on kodi there because you want to resume watching. We run in to the problem that both devices were never online at the same time and they were not able to sync the resume position. My plan was to create a new program that only implements the SyncService via platinum and just eats all changesets it gets (and puts them into some datastore) and sends them to all other devices. Since such a program would have very few dependencies, I figured it could probably run on a lot of devices like your NAS, router and what not. That way we would have a SyncService that would be able run 24/7 and we could solve that scenario above. That seems like an easier solution than ripping the gui out of kodi and trying to run a full blown kodi instance on your NAS.

[Montellese]

Can the media importer overwrite the task that is responsible to generate the changesets? In that case I could implement a 'UPnP Content Sync media importer' (which woul only differ very sligthy for the initial importing from normal UPnP servers), and I could create a specialised task that listens for changes.
It would probably be more robust to get the changesets from the source itself (onyl if possible of course) than trying to create them yourself. Image you play ItemA an DeviceA, then you play the same item on DeviceB and then they sync: Play count would be the same on both devices so the resulting play count would be one, when in fact the user has watched the item twice. Having changesets that know the playcount has been increased by one, would solve that issue (and a few more).

Well ideally the tasks trigger specific implementations that match the importer or fall back to a generic one if none is given. So in the end it should be possible to customize as much as possible but also be able to only write one part of the chain if the rest is working fine with a generic implementation.

Code:

virtual bool CanUpdateMetadataOnSource(const std::string &path) const { return false; }
virtual bool CanUpdatePlaycountOnSource(const std::string &path) const;
virtual bool CanUpdateLastPlayedOnSource(const std::string &path) const;
virtual bool CanUpdateResumePositionOnSource(const std::string &path) const;

From looking at your code, you're only planing to push changes to the source, when the 'play statistics' have changed, right?. Would it make more sense to just give the IMediaImporter implementation the whole changeset data (seems to be a 'Set<Field>'?) and the IMediaImporter can decide what he can push back to the source... An addon might not be able to push any changes or it might only be possible to send the playcount, but for the sync service we can push pretty much everything. That way, if I add a tag to one of my items, it would instantly be pushed to all other kodi Instances that are online.

I just added these as a proof-of-concept to see how difficult it would be to update stuff on the source/server. I started with these because they are probably the ones that are mainly of interest for most of the regular users. That code will be thrown away again though to be replaced by something more generic.

For now I will just focus on implementing the SyncService, since most of the changes are in libUPnP anyway. Once I have sth. actually working I will see if I can implement a UPnPContentSyncMediaImporter.

That sounds like a good plan. But you'll need the database changeset stuff for that right?

FYI: One issue I thought of was the following scenario: If you start to watch a movie in the living room, then you stop in the middle, turn it off, go to your bedroom turn on kodi there because you want to resume watching. We run in to the problem that both devices were never online at the same time and they were not able to sync the resume position. My plan was to create a new program that only implements the SyncService via platinum and just eats all changesets it gets (and puts them into some datastore) and sends them to all other devices. Since such a program would have very few dependencies, I figured it could probably run on a lot of devices like your NAS, router and what not. That way we would have a SyncService that would be able run 24/7 and we could solve that scenario above. That seems like an easier solution than ripping the gui out of kodi and trying to run a full blown kodi instance on your NAS.

Ideally one day we would be able to build an xbmc server service which basically consists of the database, and some API to access it that would run on most hardware.

I'd like to point out that nothing is set in stone yet. I've picked up where pieh left and continued working on it. Then I realized that it is very specific to the UPnP import case and wouldn't work for other sources at all. So I started to generalize the implementation. I'm sure it can be refined even more to match additional use cases. Maybe we should start by putting together a list of use cases that might make sense and once we have that we can try to design an architecture that is generic yet extendable enough to fit all these use cases.

Here are the use cases I've thought of so far:

• Import items from a UPnP ContentDirectory service. If supported also update items on the UPnP ContentDirectory service.
  
• Import items from a filesystem (basically the current "scan files into library" feature just integrated into the new framework) and scrape metadata for the items.
  
• Import specific (manually) or all items from a plugin and keep them synced (i.e. remove unexisting items and add new ones)
  
• Import recorded items from the PVR backend
  
• Import test data (for testing/debugging purposes)
  

[rterblanche]

I am watching this thread with lots of interest.
My use case is to one day have a nas/media server with no frontend handling all the scraping and managing my media.

Then for the frontend I would like to run raspberry pi's connected to my TV's in multiple room playing all media from the server.

Might have one user or multiple and would like to resume in any location/TV.

[Fice]

That sounds like a good plan. But you'll need the database changeset stuff for that right?

I think (haven't thought it through a 100%), we could get away with only storing a lastUpdate timestamp for each table column (or better each table field) and then we should be able to keep all the fields in sync, except the play count. We can't just use the absolute values especially when we want to keep more than 2 devices in sync. The specs also says, that you have to provide all the metadate of an object when you send the 'Deleted' changeset. That means we need to either just flag deleted items (instead of really deleting them) or we need to atleast keep that data around in a special table. (Although they violate their own spec in the example they give later ^^).
I think the changeset idea might be useful for other situations as well. 3rd party metadata mangers could retrieve the changesets via JSON and show some revision history to the user, perhabs with the ability to undo a change. (I don't have strong feelings for that use case though, just saying )

Use cases:
* import recorded items from the tv backend. Fallback could be the normal filesystem importer, but a backend could perhabs provide additional/initial metadata and it could add the items immediately after a recording has ended (startet?!?). And since those backends can also be media centers themself, it could be possible to sync some data with them.
* test case importer. A MediaImporter that just adds test data to the db (escpecially the edge cases).

One thing to keep in mind is, that a Source could have the same item in different qualities. Examples: UPnP Servers with transcoding or Youtube, perhabs even a tv backend that has the same chanel as sd/hd or even the same movie that's twice on disk (but that is probably more different version than different qualities).

[RockerC]

Noted this all indirectly ties into a database restructuring concept that m.savazzi trying to get going here

http://forum.xbmc.org/showthread.php?tid=200911

He is also at very early stage so might be good to get input from each other before get too far ahead