Improved allmusic.com scraper (plus a few questions)

steve1977 Wrote:Thanks, this sounds very great. Three brief questions:

1) I am not sure whether I understand your "edit". Will it be sufficient just to copy the files you have uploaded somewhere and then everything is set or do I still need to play with the settings? Given your huge music database and success of identifying artist and album art, I would like to follow your settings.

2) I have actually pretty comprehensive ID-Tags including the album cover. Does this matter for the scraper settings?

3) Do I need to activate HTBackdrops in the settings for "artist thumbs" in order to get the fanart? In other words, if I deactivate "artist thumbs from HTBackdrops", will I still get fanart from HTBackdrops?

Hey Steve!

Sorry for the delayed response, I've been away for the weekend. Hopefully you got it working In any case, I'll do my best to answer your questions:

1) The options I've added aren't enabled by default. I made that decision primarily because I thought it was best to keep the Allmusic scraper as, well, an Allmusic scraper by default, so it only scrapes Allmusic sources unless set otherwise. If you want to get better results, you need to go into the settings and disable thumb scraping from Allmusic, and enable it from Last.fm or Discogs. I'd recommend scraping artist thumbs from Last.fm and album thumbs from Discogs, though the artist thumbs from Discogs are good as well.

2) Having good ID3 tags are integral to getting good results from all the scrapers. However, as far as I can tell, XBMC prefers the album thumbs from scraper sources over the album thumbs in the ID3 tag. So if the scraper is able to fetch an album thumb from an online source, it will use that even if an album thumb exists in the ID3 tag. Maybe this isn't the intended behaviour, but it seems to be the case from my testing.

3) The "Get Fanart" is a separate setting, so as long as you keep that enabled, the scraper will still fetch fanart from htbackdrops regardless of what you set the other settings to. But FYI, I'd recommend keeping "Get artist thumbs from HTBackdrops" enabled AS WELL as getting artist thumbs from another source, because if it can't find a thumb from one source, it will automatically look to the other source. I've actually given HTBackdrops a higher priority so that it will look there first, as all the artist thumbs on that site are optimized for XBMC. But they just don't have a very large database, so it's best to have another source to fall back on.

Hope that answers your questions! Let me know how it works for you.

Thanks for this awesome scraper improvement talisto! I downloaded your files and dumped them in a recent DX build. Once I enabled the Last.fm thumbs option, I rescanned most of my library. I love the much higher quality thumbs.

One question: have your patches been added to the SVN? When I downloaded the latest SVN, I went into the allmusic scraper and the scraper options were grayed out. Of course I just replaced the scraper with the most recent one you posted in this thread and all was good!

Not a fault with your kick-ass scraper tweak, but I notice HTBackdrops doesn't find any backdrops for U2. I browsed their site and noticed that they had more than 10 HD Backdrops for U2. However, even their own site search turns up 0 results. Maybe they have a minimum search string length?

paco Wrote:have your patches been added to the SVN? When I downloaded the latest SVN, I went into the allmusic scraper and the scraper options were grayed out.

Hey paco! Glad to hear the script is working for you I've submitted a patch for my changes to be added to the SVN (nearly two weeks ago now) but so far it hasn't been approved yet. That might have something to do with the team's plan to re-organize XBMC's settings/options, I'm not sure. Hopefully it makes it in there eventually though!

You're right about HTbackdrop's search, they must have a minimum character count. I played around with it a bit but couldn't find a workaround. It'd probably be best to contact the site owner; I noticed there's no contact link on the website, but here's his forum profile, try sending him a PM: http://forum.xbmc.org/member.php?action=...&uid=28343

Sounds very nice this script, but not working here. I can see the script in the settings and I can set the new options, problem is if i go back to the settings it is default again. If an Artist is not found It get's info from another artist (always the same). Checked all the security and write options in Windows 7 and that seems ok. I'll get this with the standard Allmusic also. Do you know what is happening ?? . Firewall Also ok. Also use Latest SVN and Skin.. Happens with all the scrapers, if I search for an artist on Last FM on the internet there is lots of Info, but with the scraper it finds nothing

Roborob Wrote:Sounds very nice this script, but not working here. I can see the script in the settings and I can set the new options, problem is if i go back to the settings it is default again.

Are you changing the scraper settings from the library with the context menu (pressing "C" from the music library screen), or are you changing the settings from (Main Menu) > Settings > Music > Library > Scraper Settings? I've noticed that when you change the settings from the context menu, it only remembers the settings for one lookup and then reverts back to your previous settings. Whether this is "by design" or a bug, I'm not sure. Going through the full settings menu should work fine though.

talisto Wrote:Are you changing the scraper settings from the library with the context menu (pressing "C" from the music library screen), or are you changing the settings from (Main Menu) > Settings > Music > Library > Scraper Settings? I've noticed that when you change the settings from the context menu, it only remembers the settings for one lookup and then reverts back to your previous settings. Whether this is "by design" or a bug, I'm not sure. Going through the full settings menu should work fine though.

Thanks I did it in the context menu, I'll change them in the main menu

this one sounds very promising, cheers for that.

one (personal) question:

as you modified an existing scraper with good results, any chances you can do me a favor and use some of your time to add the functionality to look up for artist thumbs on my site?

i began to start a resource for especially prepared artist thumbs for the use in aeon (showmix) and so far 777 thumbs are present, with a bunch of users willing to add more in the future. also we started collecting fanart, but that's maybe another topic.

the gallery is organised pretty basic, thumbs are categorised in folders by artist names, so it should be pretty easy for ppl that have the skills to write a lookup to get them downloaded. every artist thumb has a thumbnail and a full picture. unfortunately there is no api that can be used, so it would be simple html scraping; but as there is an easy structure and we don't hold additional info in general, once again it should be easy to get them scraped. we also use a search feature that returns matches by given keywords in the whole gallery (thumbs AND fanart).

i would be glad if you consider trying to do that.

talisto Wrote:3) The "Get Fanart" is a separate setting, so as long as you keep that enabled, the scraper will still fetch fanart from htbackdrops regardless of what you set the other settings to. But FYI, I'd recommend keeping "Get artist thumbs from HTBackdrops" enabled AS WELL as getting artist thumbs from another source, because if it can't find a thumb from one source, it will automatically look to the other source. I've actually given HTBackdrops a higher priority so that it will look there first, as all the artist thumbs on that site are optimized for XBMC. But they just don't have a very large database, so it's best to have another source to fall back on.

Do you have prioritization built in for the artist thumbs and album thumbs such that we can just enable everything in the options but it will still try to get art in your preferred order?

Hey Azido,

azido Wrote:as you modified an existing scraper with good results, any chances you can do me a favor and use some of your time to add the functionality to look up for artist thumbs on my site?

Well, I'm a bit reluctant to do any more work on this scraper until my existing changes have been added to SVN (*if* they're ever going to be added to the SVN.. it's been over 3 weeks since I submitted the patch. I'm wondering if I've already added in more options than the dev team would prefer). But i'm also reluctant to work on features for a specific mod of a specific skin, when htbackdrops is already a fairly competent resource for general fanart; they have almost 4000 backdrops now. What is it about showmix that requires skin-specific fanart/thumbs?

Aside from that, your site would need (or should have) some tweaks before it would be scraper-friendly; artist thumbs and fanart are handled separately in XBMC so your search should have the option of returning results from one or the other. You really should consider having some sort of basic API as well; scraping HTML is somewhat sketchy, as slight changes in the design of the website can often easily break the scraper.

If/when my other changes get added to the SVN, I'll consider making you a scraper that you could at least offer as a download on your site; I doubt those options would make it into the official XBMC SVN.

fnwc Wrote:Do you have prioritization built in for the artist thumbs and album thumbs such that we can just enable everything in the options but it will still try to get art in your preferred order?

Mostly. Since the script is first and foremost an Allmusic scraper, I opted to give Allmusic the top priority for both if they are enabled, but I'd recommend turning the allmusic thumb options off to give priority to the others. If everything is enabled, the priority order for artist thumbs is (highest priority to lowest): Allmusic, HTBackdrops, Last.fm, Discogs. The priority order for album thumbs is: Allmusic, Last.fm, Discogs. There's currently no way to re-order the priority as an option.