Improved allmusic.com scraper (plus a few questions)

Should be fixed in r23269. No idea what went wrong on my first tests to reproduce this issue

vdrfan Wrote:Should be fixed in r23269. No idea what went wrong on my first tests to reproduce this issue

Awesome, thanks!! I kept testing it repeatedly and wasn't sure if I should keep pestering you about it or not

Ok, now that the music scraper settings are working properly (thanks vdrfan!), I'm back to work on creating my ultimate music scraper.

I've decided that Allmusic has the best text info, Last.fm has the best artist photos (after discovering that original/unresized photos are available through the API!), and Discogs has the best album artwork. So I've abstracted functions from all those sources, and have combined them into the Allmusic scraper as optional settings, much in the same way HTBackdrops is an option in all the scrapers.

I've also switched Discogs to use the API calls rather than HTML scraping, which I've found to be MUCH faster.

I've zipped up my music scraper folder for anyone who is interested in trying it (note: you NEED SVN r23269 or higher for this to work properly!):

EDIT: My changes are now in the latest SVN builds. Just download that instead!

Note that the Discogs functions use my own personal API key, not that it really matters, but maybe one of the XBMC devs could sign up for an "official" API key that I can add to the scrapers?

Edit: Also note, I've left the default settings to use Allmusic for everything and NOT use the other sources, so you'll have to go into the Scraper Settings if you want to enable scraping from Last.fm and/or Discogs.

Sweet. Trac it please.

Btw, I've played with the discogs API myself some time ago but dropped it, as it seemed to be a PITA to identify the correct album. Searching for "Kelly Clarkson" (no, not my music taste, just an example) returned a Disney Single in first place eg.

vdrfan Wrote:Btw, I've played with the discogs API myself some time ago but dropped it, as it seemed to be a PITA to identify the correct album. Searching for "Kelly Clarkson" (no, not my music taste, just an example) returned a Disney Single in first place eg.

The artists search seems to work alright, as you can specify type=artists, e.g:

http://www.discogs.com/search?type=artis...7404bc20ec

The problematic part is the album search, you can specify type=releases, but you can't separate the artist and album into separate variables, so you need to mash them together like this:

http://www.discogs.com/search?type=relea...7404bc20ec

Which can result in theory some false positives, but in my testing I've found it to be pretty reliable.

The head-scratcher is that the API information doesn't seem to always match the HTML version. For instance, look at the HTML page for Beck:

http://www.discogs.com/artist/Beck

Then look at the API version:

http://www.discogs.com/artist/Beck?f=xml...7404bc20ec

Notice the API doesn't have the biography/profile? It DOES for some artists, but not for others. There's a lot of inconsistencies in that regard. So I think it's useful in the way I'm using it, since I'm only using it for images, and it's much faster than parsing the HTML. But I'm not so sure it's a good replacement for the stand-alone Discogs scraper.

vdrfan Wrote:Sweet. Trac it please.

Done!

http://trac.xbmc.org/ticket/7254

Quote:Edit: Also note, I've left the default settings to use Allmusic for everything and NOT use the other sources, so you'll have to go into the Scraper Settings if you want to enable scraping from Last.fm and/or Discogs.

Forgive me if I'm being dense... but where are the music scraper settings? I can get them from all the video sources, but don't seem to have access to anything but simply picking the scraper in the music settings.

I know I'm missing something obvious...

blacklist Wrote:Forgive me if I'm being dense... but where are the music scraper settings? I can get them from all the video sources, but don't seem to have access to anything but simply picking the scraper in the music settings.

Well, first, you need to have a VERY recent SVN version of XBMC. The scraper settings were only added a couple weeks ago. If you do have a recent SVN version, you can access the scrapers by going to Settings > Music > Library > Scraper Settings. If you don't have that option, you need to update your version of XBMC.

BTW, I nuked my whole music library and rescanned all my files (admittedly a meager collection of roughly 10,000 files; 800 albums and 1000 artists) with my new scraper update, using Last.fm for the artist thumbs and Discogs for the album thumbs (all other settings disabled except for HTbackdrops), and I must say, I'm REALLY happy with the results! Nearly every artist has a high-resolution thumbnail and a full bio, and nearly every album does as well, with no false positives that I can see (yet).

Not to toot my own horn, but I really do think this covers all the bases better than expected. Most of the "glory" of XBMC goes to it's video-handling capabilities, but for those of you out there who use it primarily for music like I do, I think you're going to be really pleased with this once it hits the SVN.

talisto Wrote:Ok, now that the music scraper settings are working properly (thanks vdrfan!), I'm back to work on creating my ultimate music scraper.

I've decided that Allmusic has the best text info, Last.fm has the best artist photos (after discovering that original/unresized photos are available through the API!), and Discogs has the best album artwork. So I've abstracted functions from all those sources, and have combined them into the Allmusic scraper as optional settings, much in the same way HTBackdrops is an option in all the scrapers.

I've also switched Discogs to use the API calls rather than HTML scraping, which I've found to be MUCH faster.

I've zipped up my music scraper folder for anyone who is interested in trying it (note: you NEED SVN r23269 or higher for this to work properly!):

http://www.mediafire.com/download.php?bxdvj1jzntz

Note that the Discogs functions use my own personal API key, not that it really matters, but maybe one of the XBMC devs could sign up for an "official" API key that I can add to the scrapers?

Edit: Also note, I've left the default settings to use Allmusic for everything and NOT use the other sources, so you'll have to go into the Scraper Settings if you want to enable scraping from Last.fm and/or Discogs.

Thanks, this sounds very great. Three brief questions:

1) I am not sure whether I understand your "edit". Will it be sufficient just to copy the files you have uploaded somewhere and then everything is set or do I still need to play with the settings? Given your huge music database and success of identifying artist and album art, I would like to follow your settings.

2) I have actually pretty comprehensive ID-Tags including the album cover. Does this matter for the scraper settings?

3) Do I need to activate HTBackdrops in the settings for "artist thumbs" in order to get the fanart? In other words, if I deactivate "artist thumbs from HTBackdrops", will I still get fanart from HTBackdrops?


Cheers!!!