[Release] Universal Scraper for Music Artists

No, please add those images to fanart.tv, so you give something back to the community.

I just added my music collection to XBMC this weekend and played around with the music artist scrapers. First off thanks for developing these useful add-ons.

Here is what I found. I initially just used the Universal Scraper at its default settings. I found that it was incomplete with discography information and artist bios, sometimes it got it sometimes it didn't. Also thumbs and fanart was sometimes there, sometimes not. What I had to do was then re-scan my entire library (remove source, re-add source) with the last.fm artist scraper and then later the musicbrainz scraper to pull in some of the missing information to my cache and then finally re-run the Universal Scraper to get most of the information to appear and be available for all artists (still some gaps but much better).

Given that the Universal Scraper is set up to scrape information from Last.fm and similar information on thumbs and fanart as in those other scrapers I am not sure why it didn't in many cases and I needed to ultimately run all three scrapers to get information into cache.

Maybe I'm doing something wrong? I am definetly brand new to XBMC. But I found much greater success and having complete artist information by running all of them.

sometimes is not something i can test.

(2012-06-04 14:39)olympia Wrote:  No, please add those images to fanart.tv, so you give something back to the community.

Thanks for your replie.

(2012-06-04 18:35)olympia Wrote:  sometimes is not something i can test.

Understood. Just an overall comment then. My running of the universal scraper returned poor results overall. I needed to run last.fm and musicbrainz to grab information that I had thought would have been scraped by the the universal scraper. Even though the universal scraper is set up to grab artist bios from last.fm, it did not and I would need to run last.fm to get those bios as an example.

Dear Andy,

I am absolutely not interested in overall comments unless those are positive.
In case you have a problem or complaint you need to be very specific! "Sometimes there, sometimes not" doesn't help at all.

I see that you are new to the forum and you mentioned your are brand new to xbmc, but this is not a good excuse to share your unqualified feedback which might mislead others. E.g. Musicbrainz scraper and Univeral Scraper can't give different results at all since the Universal Scraper is just an extended Musicbrainz scraper (which in fact doesn't provide a lot in its own).

There can be several reasons why last.fm and Universal scrapers give different results and I would be interested to look at this.
So let me re-articulate my request: _give me an artist to test_.

(2012-06-04 19:40)olympia Wrote:  Dear Andy,

I am absolutely not interested in overall comments unless those are positive.
In case you have a problem or complaint you need to be very specific! "Sometimes there, sometimes not" doesn't help at all.

I see that you are new to the forum and you mentioned your are brand new to xbmc, but this is not a good excuse to share your unqualified feedback which might mislead others. E.g. Musicbrainz scraper and Univeral Scraper can't give different results at all since the Universal Scraper is just an extended Musicbrainz scraper (which in fact doesn't provide a lot in its own).

There can be several reasons why last.fm and Universal scrapers give different results and I would be interested to look at this.
So let me re-articulate my request: _give me an artist to test_.

I shared my experience and results with the Universal Scraper. I had poor results. Sorry it offended you but I thought it might be helpful to others to share what I experienced and the steps I took to get artist information somewhat properly populated.

For me there were many artists that had missing information and my focus was getting my large music collection in decent shape in XBMC and properly scraped. It took me running the universal scraper, last.fm, musicbrainz and then universal scraper again one last time to get it there.

I didn't document and track things artist by artist. There were many. To recreate the results for testing purposes I would have to completely wipe everything, including all cache if I knew where everything is stored, and re-run to see precisely all the artists where information was missed. It took so much time just to get things where they are right now I'm not going to retrace my steps. But I will focus artist by artist for future additions and provide you those names.

Great job on the scraper, it works very nicely! A few points of discussion, I'm very new to XBMC so let me know if I've got the wrong end of a stick here...

• IMHO,using MBID's is essential for fetching accurate data. There are many artists that have the same/very similar names.
• Most artists (and albums) on Musicbrainz have a link to the Wikipedia page attached, this could be used to fetch bio information.
• Google should also be added as a source for images, I understand that people should contribute images to fanart.tv but that is very time-consuming (plus I have contributed countless hours to editing MB) and finding images is something that google is already very good at.
• Discogs is another very good source of music metadata that would make a good addition to this scraper.
• I would also love to see a Universal Scraper for albums, many possibilities there.
  

Thanks, keep up the good work.

(2012-06-06 07:29)saladasalad Wrote:  [*]IMHO,using MBID's is essential for fetching accurate data. There are many artists that have the same/very similar names.

This is not possible currently (MBID is not scanned in, stored and made available to the scrapers for lookup), but this is something actively worked on atm: http://forum.xbmc.org/showthread.php?tid=132573

(2012-06-06 07:29)saladasalad Wrote:  [*]Most artists (and albums) on Musicbrainz have a link to the Wikipedia page attached, this could be used to fetch bio information.

I will look at this, but ot high on my prio list.

(2012-06-06 07:29)saladasalad Wrote:  [*]Google should also be added as a source for images, I understand that people should contribute images to fanart.tv but that is very time-consuming (plus I have contributed countless hours to editing MB) and finding images is something that google is already very good at.

No way at all. This is a scraper for top quality.

(2012-06-06 07:29)saladasalad Wrote:  [*]Discogs is another very good source of music metadata that would make a good addition to this scraper.

What Discogs has which is not available on MusicBrainz? I am really not a fan of discogs, but I can be convinced.

(2012-06-06 07:29)saladasalad Wrote:  [*]I would also love to see a Universal Scraper for albums, many possibilities there.

This is in the works now.
However I don't see the many possibilities there. Hints are welcome. (Actually I don't see so much sense to be able to scrape tracklist from 5 different site if it will be the same). Having different sources IMO only make sense if the data we scrape is different (like review, ratings, artwork).

(2012-06-06 09:22)olympia Wrote:  

(2012-06-06 07:29)saladasalad Wrote:  [*]IMHO,using MBID's is essential for fetching accurate data. There are many artists that have the same/very similar names.

This is not possible currently (MBID is not scanned in, stored and made available to the scrapers for lookup), but this is something actively worked on atm: http://forum.xbmc.org/showthread.php?tid=132573

I see. Thanks for the link, I'll have a good read through that later. Sounds promising.

(2012-06-06 09:22)olympia Wrote:  

(2012-06-06 07:29)saladasalad Wrote:  [*]Google should also be added as a source for images, I understand that people should contribute images to fanart.tv but that is very time-consuming (plus I have contributed countless hours to editing MB) and finding images is something that google is already very good at.

No way at all. This is a scraper for top quality.

Fair enough. I do think that with the right settings (high image quality, -covers, photos, etc.) you could get decent quality but it's probably something better suited to another addon.

(2012-06-06 09:22)olympia Wrote:  

(2012-06-06 07:29)saladasalad Wrote:  [*]Discogs is another very good source of music metadata that would make a good addition to this scraper.

What Discogs has which is not available on MusicBrainz? I am really not a fan of discogs, but I can be convinced.

Nothing particularly useful on an artist level now I think about it, other than some images (which are probably on last.fm anyway.) Discogs does usually have more album releases than MB though, unfortunately more people seem to contribute to Discogs than MB. I'm not a fan of the site either, TBH.

(2012-06-06 09:22)olympia Wrote:  

(2012-06-06 07:29)saladasalad Wrote:  [*]I would also love to see a Universal Scraper for albums, many possibilities there.

This is in the works now.
However I don't see the many possibilities there. Hints are welcome. (Actually I don't see so much sense to be able to scrape tracklist from 5 different site if it will be the same). Having different sources IMO only make sense if the data we scrape is different (like review, ratings, artwork).

Good to hear. Many possibilities was an exaggeration on my part but there are certainly possibilities . Multiple reviews from metacritic for each album. Ratings from rateyourmusic, metacritic or MusicBrainz. You could also get release metadata like label, cat# and release date. Album specific Wikipedia links for album descriptions. That's all I've got right now but I'm sure I'll think of more, I've just discovered XBMC recently and I'm brimming with ideas but I need to write them down as I think of them or something