Allow artist scraping from more than one service (music section)

Jonathan,

Fair enough, if by "crap out badly" you mean "has info, but it's wrong or isn't very good" -- is that a common case? It usually seems to be a completeness (rather than quality) issue for me (so I'd love to see a system default even if there are per-source overrides), but I'd be a fool to define the world by my own limited experience. Now that I think about it, you might be more network-efficient even in the completeness case if you knew that a particular scraper was more likely to have the data associated with a particular source.

Do you think that this same paradigm might apply to the music side of things as well? I've never really thought about splitting music across multiple sources, but I could certainly imagine people doing it...

I do really think that some kind of user-prioritized scraper chaining (or merging) makes a heck of a lot of sense, though, so I'll happily wait to see what comes out of GSoC...

Keep up the good work, and thanks for responding so quickly!

It's both basically - completeness is no problem to take care of (as you can simply replace it with info from the next in the chain), but rubbish data (or data that is not very good) is also an issue. A simple example would be themoviedb's ratings vs imdb's ratings (or rotten tomatoes ratings etc) where all have the data, but some are better (to the user at least) than others.

If we could get each scrape source to give a rating of what is decent data and what isn't, then we could prioritise individual bits of data from each scraper reasonably well. Ofcourse, user config will always be required, but hopefully it won't need to be too fancy (or everything will work well enough for most users that they don't need to care about it.)

Cheers,
Jonathan

True, some kind of quality metric makes sense. Thinking about implementation, though -- how would you (the general, plural, "you", here) envision the quality rating info for scraper data to be acquired? There's no way under the blue sky that all of the data providers will add that kind of meta-metadata. I guess you could break down the metadata into subclasses and have community or personal quality ratings for that (e.g. scraper A has good ratings, while scraper B has good thumbnails, etc.)

There might be a way to get date/time information for the metadata -- perhaps equating "newer" with "better" is a reasonable rule of thumb...

Maybe the core doesn't really need to worry about it at all, though. Between CU Lyrics going to chained sources and the Universal Scraper that's all over the forums, it looks like the addons themselves are taking up the charge and handling chaining, merging, etc. I get a little worried about the addons doing that, though, as it generates a more fragmented interface, so I'd like to see something from the core.

(2012-06-04 15:37)PartialGestalt Wrote:  True, some kind of quality metric makes sense. Thinking about implementation, though -- how would you (the general, plural, "you", here) envision the quality rating info for scraper data to be acquired? There's no way under the blue sky that all of the data providers will add that kind of meta-metadata. I guess you could break down the metadata into subclasses and have community or personal quality ratings for that (e.g. scraper A has good ratings, while scraper B has good thumbnails, etc.)

There might be a way to get date/time information for the metadata -- perhaps equating "newer" with "better" is a reasonable rule of thumb...

Maybe the core doesn't really need to worry about it at all, though. Between CU Lyrics going to chained sources and the Universal Scraper that's all over the forums, it looks like the addons themselves are taking up the charge and handling chaining, merging, etc. I get a little worried about the addons doing that, though, as it generates a more fragmented interface, so I'd like to see something from the core.

Instead of thinking on how to use multiple scrapers why noy just improve the content of the sources sites?
If everyone just does it's part of adding the missing info everyone can benefit.

Besides that there are some new scrapers available which you can choose the order of the scraper info. This is what you want. However i still ask everyone to do his part in adding info

(2012-06-04 15:40)Martijn Wrote:  Instead of thinking on how to use multiple scrapers why noy just improve the content of the sources sites?
If everyone just does it's part of adding the missing info everyone can benefit.

I agree wholeheartedly, and while it is undoubtedly true that everyone benefits from improved and updated metadata, and we should all update the sources we use with the information that we know, we can't rely on that to fully solve the problem, since it would require that every metadata source have all of the data at an equivalent quality as every other source. Most folks aren't going to add their updates to IMDB and TMDB and other favorite movie site, etc; they won't update MusicBrainz and AllMusic and Last.FM. They'll pick their favorite and add the data there. The ideal is that everyone adds the info, and "everyone" is a large enough set, that it'll work perfectly for whatever site is your favorite. The reality is, and probably always will be, that the available metadata from any single site is not as good as union of data gleaned from chained or merged accesses to multiple sites.

(2012-06-04 15:40)Martijn Wrote:  Besides that there are some new scrapers available which you can choose the order of the scraper info. This is what you want. However i still ask everyone to do his part in adding info

Yep, and that's what I was talking about in the previous post. The presence of this kind of aggregate data scraper really only proves the point that users want to be able to get combined data. Those scraper authors are fantastic and talented, and I have nothing but appreciation for their time and efforts and results, and if they end up being sufficient, then wonderful -- we all win. It seems to me like the overall user experience would be better if that kind of data aggregation was handled in the core, though, since the config and selection UI would be consistent across the data types. It would avoid a situation where one datatype is purely priority-based while another is fully merged, while another is handled some other way, etc.

How did GSoC 2012 go? Was this addressed?