Allow artist scraping from more than one service (music section)

Curently in the settings I can choose one default service to look for artist info (thumbnail, fanart, info, ...). Allmusic is the default service, but sometimes it does not provide an artist picture and there is no way to edit an artist entry on allmusic. So it might be good to search on freebase.com or last.fm. But then I'd have to switch the default scraping service.

Wouldn't it be smarter to simply allow the user to setup the services in order, so that the first service would be the priority service to check. If certain info is not found on that service move on two the second entry and check there.

That would be a much more efficient method to get this job done.

Maybe this is already possible (then I'd be very happy about a "how-to" for noobs).

Cheerios,
sync

PS: How can I auto-scan all artists in my library? Having to go through each artist manually is a waste of time. Didn't we build computers to do stupid tasks like that, in the first place?

How many services do you want? If order takes priority, what happens if the first one contains useless information? How does the user then switch to the next one?

Think through how it would work from a user perspective (what would the UI look like?)

(2012-03-25 16:42)Syncopation Wrote:  PS: How can I auto-scan all artists in my library? Having to go through each artist manually is a waste of time. Didn't we build computers to do stupid tasks like that, in the first place?

Hi. earlier we were talking about this in IRC. I am happy to say that I found out how to do that.

1. Go to System -> Settings -> Music -> Database
2. Select "Grab additional information when refreshing database" (I am losely translating from the german UI)
3. Refresh your Database
4.
5. Profit

@jmarshall: In my case I'd like to have the following services in following order:

a) allmusic
b) last.fm
c) freebase.com

allmusic has the most solid database but sometimes artist pictures are missing and I can't edit their database. So I'd use b) or c) for that. So the order would then be determined by the oder I give the scrapers in the settings view (currently not possible). Then if no artist picture is found XBMC would move to the second scraper and try if it can find an artist picture at there etc. pp. The user would not switch. All this would be done automatically. Simply by using the scrapers order as priority and is fields remain empty xbmc moves to the next scraper on the list.

Currently I don't really understand the situation. I used allmusic first. And it seems that more than one scrapers are not allowed.

@Horscht: tired your suggestion like this: changed the setting you mention. let allmusic run. many missing information components. switch to last.fm and re-run, but it seems many item which have half info get skipped. So not really useful.

Still unsure how to deal with this and think this could be done better by allowing 2 or 3 scrapers. (Doesn't plex work like this? - if information is not found move to the next service?)

Hi,

I'm proposing some scraper changes and one of them would allow just this kind of modularity ( didn't have the fallbacks option in mind yet but would be an easy extention ). See here: http://forum.xbmc.org/showthread.php?tid=126365 You can always post your request there too then I'd include it in my proposal when writing. So it's up to you to convince jmarshall to pick me

Hey Sander!

Thanks for linking this existing thread. I wish I could convince jmarshall of this, but he has not responded to my newest post yet, so I don't know if there are any counter arguments so far - at all.

Hi everybody,

actually I'm new to XBMC and I'm thinking to use it as my new media backend now. Fact is, I like XBMC a lot although I still have some problems in understanding how the GUI is working . I'm coming from "Music Section: Artist Info (how to scrape artist info automatically)" and I just experienced a strange problem when I wanted to add thumbnails for all my artists in the library.

I'm using Eden os OSX 10.6 and my music is stored on a NAS which I access via SAMBA. Unfortunately, I have some AFP issues for which I do not have time for solving at the moment. The music is organized in a folder containing folders for every artist which contain folders for the albums. I managed (after reinitializing the library and doing some "random" clicking) that the scrapers dowloaded the album covers. Now I would like to have those fancy artist infos and fanart like in http://wiki.xbmc.org/index.php?title=Fil...t.WSCR.jpg So I went into the library - artist section and opened the context menu. I was having only those generic thumbnails and there was no option to load all artist informations. So I clicked "artist information" and then the window to _enter_ the name of the artist opened - no artist information. Actually clicking on "done" put me into an infinite loop - I was brought back to enter the name of the artist. Hitting escape slved the problem. Then I tried to change the scraper (I used the last.fm one), and suddenly when opening the context menu of one artist the optiion to "load _all_ artist informations" showed up. Nevertheless, when clicking that option only the information for the selected artist was downloaded and after that this option vanished again. So I changed the scraper back to AllMusic and the option reappeared. I tried this several times and I can reproduce it - I always have to switch the scraper before I can download one new artist thumbnail. When opening the context menu of an artist which has already the informations I have the option to load all artist informations but it doesn't load anything.

Either I'm doing something completely wrong or there is a problem in the way the scrapers work.

Thanks in advance,

Ben

Hi everyone,

I'm facing the same problem as Meow, I need to switch scrapers everytime I want to download a new artist information. Its very annoying. Is there a solution/fix for this?

I would put myself in this camp, as well, although I would extend the idea to all the scrapers; it's annoying to change scrapers and rescan, not to mention the cumbersome process of changing each source's settings when you switch to a new preferred scraper.

To answer jmarshall's final question above, here are two different visions for the UX (my aplogies for the length, but I'd like to be as complete as possible):

First option: lower flexibility, easier implementation (call this the "FST" (i.e. "first","second","third") option):
1. Consolidate scraper options into a single location (central, not per-source). This is currently done (as of Eden) for music (System->Music->Library->Default ... service), but video scrapers are tied to individual sources.
2. That central location would be a "Metadata Sources" panel in the system settings, with a section for each scraper type. In a Confluence-style skin, this would mean a left-menu containing "Artist Info","Movie Info", etc., with the right-side list being the ordered list of scrapers.
3. Implement the "ordered list" as 3 settings: "First Scraper", "Second Scraper", and "Third Scraper", where each setting is an enum of the installed scrapers of a given type, plus 'none'.
By using the enum and fixed settings for "first", "second", and "third", we keep using all the same widgets, and internal code churn is kept to a minimum.

Second option: full flexibility with new setting/widget type (call this the "new-widget" option)
1. As with the previous option, I'd collect the scraper preference settings into a common location under the main settings
2. Add a new setting type "orderedenum" (or something similar) that pops up a list when selected; the ordered list would be populated by the set of currently-installed scrapers of the appropriate type, and the user could move entries up or down the list using the same mechanisms that are in place for the "Now Playing..." window.
By using the new setting/widget type, we allow all scrapers to come into play, and avoid stale settings if a scraper is uninstalled. There would be nontrivial internal (and skin) churn in implementing the new setting type, but it would be a fairly straightforward effort, and the results would be useful in more cases than just prioritized scrapers.

Now, the use cases:
1. Music
Music hardly changes. Currently the user goes to the main music library settings and configures a default scraper. This behavior would be preserved, save the fact that the user picks a sequence, rather than a default, scraper.
2. Video
When adding a video source, the user would only select a source type, rather than source type+scraper. This actually simplifies the process of adding or editing a video source, since it removes a chunk of complexity from the edit source window.
To set the scraper preferece, the user would go to the system settings instead of editing each source. I believe that this is more intuitive, since this makes video scraper selection the same as music scraper selection, and consistency is _good_ for UX.

Adding/removing scrapers:
1. When installing a scraper, it is immediately added to the lowest-priority slot on the list of that type. For the FST option above, this means adding it to the highest not-set option (or not at all if all three have been user-selected). For the new-widget option, this means adding it to the bottom of the list.
2. When uninstalling a scraper, it is simply removed from the list. For the FST option, this may mean shifting up (i.e. the second choice becomes the first choice). For the new-widget option, the entry is just removed.

Scraping data:
When scraping data, the system would do much like it does now -- pick the preferred scraper and attempt to fetch the data. The difference would be how it handles a "not found" failure -- previously it gave up, but now it would go to the next-most-preferred scraper and repeat the attempt.

To answer a few more of the original questions:
Q1. "How many services?"
A1. I'd say at least 3, although ideally it would be "however many are installed"
Q2. "If order takes priority, what happens if the first one contains useless information?"
A2. If it contains useless information, you get that useless information. This is how it works now. If you don't trust a scraper's data, put it lower in the priority list.
Q3. "How does the user then switch to the next one?"
A3. The user doesn't. If a scraper fails (can't contact server, entry not found, etc), the system falls through to the next scraper in the list. The user expresses a preference in the settings, and the system implements those preferences.

There are a number of other changes that could be implemented (e.g. merged data from scrapers), but a simple failover across a user-prioritized list would seem to satisfy the bulk of the need.

The only problem there is that tv source "A" needs scraper 1 whereas tv source B will crap out badly with scraper 1 but will do awesome with scraper 2. Prioritising 1 over 2 means tv source B has bad data. Thus, the per-source selection. Now, it may well be that this can be solved by having a more refined source type (tvshow split into a bunch of different genre types for example), or by allowing the source to be tagged in some way (or allowing a different order of scraping for that source...)

Hopefully some of this will be addressed with topfs2's GSoC this summer, by allowing both better chaining/multiple runs of scrapers, and allowing scrapers to feed back accuracy information.

Cheers,
Jonathan