Unified music addon frontend

I would still say that if we want a spotify addon anytime soon the way to go is to allow binary addons with a certain structure and add hocks like:

Albums -> get albums from local db and ask all music addons if they have the ability to provide albums and if they do receive them and show them mixed with the local albums.

In that case spotify (in my opinion) should return all albums that has all of its tracks "stared".

A cache would add some additional problems like promontis says, for example spotify allows playlists to contain tracks that are not showing up if you browse tracks (stared tracks). Thats no problem if you just ask the plugin for the playlists, but if they are to be stored in a db its not going to work.

Anyway, we should come to a conclusion on how this should be implemented so that we can start the actual work. I have most parts of a new fast spotify backend coded but I'm saving the XBMC close coding until we now how we want it.

/David

Right, I think we're on the same basic track then. XBMC will ask the addon "give me this type of content please, possibly using this filter".

If the addon can do the filtering, it will give us a filtered list. If not it will give us everything and we'll do the filtering. Ideally it would also tell us how persistent the data is as well (eg this data valid for next 24 hours).

We can then cache the results so that next time XBMC wants the same thing we don't have to hit the addon necessarily (or we could return the cached results while hitting the addon for updates).

Type of content might be "Album, Artist, Song, Movie, TVShow, Episode, Playlist??" and perhaps some more general "Collection".

Idea being that the addon can operate in directory browsing mode, whereby it throws back collections of types of content, and it can also operate in a more direct search mode if it can handle filtering.

User clicks on "Bob Dylan" and XBMC retrieves albums in the local collection that match, and also retrieves tracks/albums/collections from addons that also match.

How we end up showing these in the UI can then be decided. One option perhaps is just showing a virtual folder like "Spotify playlists with Bob Dylan", or alternatively showing the spotify playlists with Bob Dylan in the list directly etc.

Cheers,
Jonathan

jmarshall Wrote:Right, I think we're on the same basic track then. XBMC will ask the addon "give me this type of content please, possibly using this filter".

Just want to say that any of those, with our without filtering can take several minutes on uPnP. But could work around that with a cache in the addon. But then its cache on cache, so might make sense to just simply assume that addon is fast (or have cache)?

jmarshall Wrote:If the addon can do the filtering, it will give us a filtered list. If not it will give us everything and we'll do the filtering. Ideally it would also tell us how persistent the data is as well (eg this data valid for next 24 hours).

I find this a bit backward The client has no way to predict the future, I mean I would get events signalling when something changes on a uPnP MediaServer but I have no way to tell how long the current database is valid, it can be for one second or a whole lifetime.

jmarshall Wrote:We can then cache the results so that next time XBMC wants the same thing we don't have to hit the addon necessarily (or we could return the cached results while hitting the addon for updates).

Type of content might be "Album, Artist, Song, Movie, TVShow, Episode, Playlist??" and perhaps some more general "Collection".

I'd rather either assume that the backends could be terribly slow or they are as fast as our db (with or without a cache whatever they need).

jmarshall Wrote:Idea being that the addon can operate in directory browsing mode, whereby it throws back collections of types of content, and it can also operate in a more direct search mode if it can handle filtering.

User clicks on "Bob Dylan" and XBMC retrieves albums in the local collection that match, and also retrieves tracks/albums/collections from addons that also match.

How we end up showing these in the UI can then be decided. One option perhaps is just showing a virtual folder like "Spotify playlists with Bob Dylan", or alternatively showing the spotify playlists with Bob Dylan in the list directly etc.

Cheers,
Jonathan

I guess we could just have the backends be a glorified plugin, I mean just a plugin but with strict folder hierarchy. It would be simple to implement and should work quite well. AFAICT thats almost what you are suggesting right?

The point is that it shouldn't be up to the addon to implement the cache. If they're fast they don't have to worry about it - just send us everything.

If they're slow then we can do the caching for them. We can always hit them no matter what.

But yes, IMO all we really need is a simple interface with pretty much 2 API calls: Give me a listing (id and filter) or perform a search (search term, filter).

Cheers,
Jonathan

jmarshall Wrote:The point is that it shouldn't be up to the addon to implement the cache. If they're fast they don't have to worry about it - just send us everything.

If they're slow then we can do the caching for them. We can always hit them no matter what.

IMO there should simply be a "dirty" flag for each list. When a list is retrieved, it checks the addon if this flag is set, which in that case, it reloads the list. Otherwise, the cached list is being used. Most systems have some single method of checking whether the content of a list has changed or not; spotify is atleast one of them (eg. playlist_added/removed, etc)

promontis Wrote:IMO there should simply be a "dirty" flag for each list. When a list is retrieved, it checks the addon if this flag is set, which in that case, it reloads the list. Otherwise, the cached list is being used. Most systems have some single method of checking whether the content of a list has changed or not; spotify is atleast one of them (eg. playlist_added/removed, etc)

That won't really work well with uPnP. While I know when a list is dirty it may take me a minute to get the new list (depending on the server) and I'd rather not do that when GUI asks for list.

jmarshall Wrote:The point is that it shouldn't be up to the addon to implement the cache. If they're fast they don't have to worry about it - just send us everything.

If they're slow then we can do the caching for them. We can always hit them no matter what.

But yes, IMO all we really need is a simple interface with pretty much 2 API calls: Give me a listing (id and filter) or perform a search (search term, filter).

Cheers,
Jonathan

It would work if a) xbmc asks us for list at startup and b) that we can give the result in an async fashion.

a) since it may take me several minutes to produce the list and b) because the list may be thousands of songs and introducing them as one big chunk is kindof bad IMO.

I'm fine with having cache in addon and have GUI always aggregate from us, infact we can hide the cache in the wrapper. But I think its very bad to do a middle ground where both caches (which seems needed with some of the suggestion put forward here )

I think it would work well if we do one of the following:

• XBMC asks for filtered info as needed and aggregate over many sources. e.g. it asks for all artists, all tracks of X artist etc. i.e. Addon keeps cache if needed
• Addon provides cachable info (xbmc could ask for it at startup also) in an async fashion. e.g. The addon states that track X is available when it finds it, the GUI simply uses the tracks found at the time of listing when listing. This can be achieved by either having cache in addon wrapper OR big cache for all addons
  

Both of those ought to work with upnp and spotify afaict and I'm behind any of them

topfs2 Wrote:That won't really work well with uPnP. While I know when a list is dirty it may take me a minute to get the new list (depending on the server) and I'd rather not do that when GUI asks for list.

So a dirty flag AND getting a new list on start-up (idle)? Will that work? Of course, everything should work in an async matter.

promontis Wrote:So a dirty flag AND getting a new list on start-up (idle)? Will that work? Of course, everything should work in an async matter.

Well as said I know when something changes quite immediately, so I could mark dirty real quick. But refreshing the cache when asked may take up to a minute or so. But if we assume xbmc refreshes it immediately when I mark dirty and doesn't wait until GUI wants it then that would work.

Somehow it seems easier for me to when I get the event to just start sending UpdateTrack(X, new track) type of async stuff instead of waiting for xbmc to decide when I should update the cache Afterall I (the addon) knows when and what changes, seems sane that I'm in control of it? But, whatever is easiest to implement for all parties seems good.

topfs2 Wrote:Well as said I know when something changes quite immediately, so I could mark dirty real quick. But refreshing the cache when asked may take up to a minute or so. But if we assume xbmc refreshes it immediately when I mark dirty and doesn't wait until GUI wants it then that would work.

Somehow it seems easier for me to when I get the event to just start sending UpdateTrack(X, new track) type of async stuff instead of waiting for xbmc to decide when I should update the cache Afterall I (the addon) knows when and what changes, seems sane that I'm in control of it? But, whatever is easiest to implement for all parties seems good.

Even better! So basically each addon "pushes" its changes to XBMC. That seems like the easiest and fastest solution for me.

Been reading your post and I find what you are doing to be super interesting and totally ambitious. Just as a point of interest, I thought this might be a starting- off point, at least as far as how the scraping could work. bradvido88 has made a "My Library" script that seems to be able to scrape video data from most sites that have plugins and adds the videos to your personal library: http://forum.xbmc.org/showthread.php?tid=98210

I know you are focusing on music and spotify for now, but at least this proves that it is possible to add streaming sources to the local library. I have not tried it myself, but I intend to test it out one of these days I have sufficient free time.