Last.fm scraper in development - help wanted
#16
that would be neat, and some system to cross reference several scrapers in order to make matches more reliable
Reply
#17
AllMusic.com isn't updated on a regular basis and doesn't list many relatively unknown artists. I'd expect a boost in search results from almost any new scraper, if its last.fm or discogs.. These two sites are both as good, provide a decent API etc.. One might be better than the other when it comes to specific musical nices. Most people are better off to use their favorite scraper for regular searches. We might want to make it a bit easier to switch scrapers, from the Album/Movie info screen or the manual search screen maybe ?

Anyway, in meantime I did some testing on a discogs.com scraper. This scraper script is just a few lines, it has CreateAlbumSearchUrl and GetAlbumSearchResults nodes and thats about it. Where I get stuck is testing the thing, XBMC simply crashes if I use it (no messages in xbmc.log).
The 'Scrap' test tool looks promising, however it is built to handle video scrapers, not music. Tried to fix that as well, but got caught up in C++ syntax :S Scraper development would be much easier if anyone could fix that Scrap tool. Doesn't look like an easy job, though ..
Am I missing a development tool here, should I be debugging from Visual C++ etc. ?
Reply
#18
kriziz Wrote:Where I get stuck is testing the thing, XBMC simply crashes if I use it (no messages in xbmc.log).
The 'Scrap' test tool looks promising, however it is built to handle video scrapers, not music. Tried to fix that as well, but got caught up in C++ syntax :S Scraper development would be much easier if anyone could fix that Scrap tool. Doesn't look like an easy job, though ..
Am I missing a development tool here, should I be debugging from Visual C++ etc. ?

I was fighting with the same thing last night. The 'Scrap' tool doesn't work correctly. Getting it to compile and run on Linux will be the first step, so maybe I'll try to get that sorted out over the upcoming weekend. Trying to debug scrapers through XBMC's GUI doesn't sound like fun.
Reply
#19
scrap is broken as we lost the source (blame donj for bad committing practices).

the way i do scrapers is using a regexp tool + printf. it works just fine for me, but an updated scrap tool would be invaluable for sure
Reply
#20
oh damn. kriziz, i have a semi-done discogs scraper (only album part)! we should try not to dupe work Smile

if you want i'll gladly leave the artist part to you Smile
Reply
#21
basic scraper added to svn
Reply
#22
not even a 'it does not work you silly sod'?
Reply
#23
eh Smile
i tried it but it didn't retrieve much... does it need a last.fm account set up?

(this was a politically correct version for 'it does not work you silly sod' :p )
Reply
#24
shouldnt require anything liek that no.

last.fm offers very little info. just artist, album name, year, releasedate, review and track listing.

even less for artists, although some of the biography's are good.

you did not get any info at all?
Reply
#25
really i didn't try it much... i'm going to try again and see, thanks Smile
Reply
#26
Thanks a lot spiff, I don't have the time to check it now, but this is one of the features I was really looking forward for XBMC to have Smile
Reply
#27
Would you be able to scrape each tracks 'listeners' and apply it to the 'ratings' field of the database?
each album might need to be calculated relative to itself
Reply
#28
i have absolutely no idea what that last sentence is supposed to mean.

http://www.last.fm/api

as you can see there is no way to search for tracks from a specific album
Reply
#29
Thumbs Up 
Hey spiff,

i tried out your last.fm scraper and must say it works brilliant. The only downside are the crappy artist/album pics from last.fm but that is not your fault.

It would be nice if some kind of a language selection would be possible. Perhaps with a fallback to english if no information for the selected language is availabe.

Thanks for your great work

TheNME123
Reply
#30
TheNME123 Wrote:Hey spiff,

The only downside are the crappy artist/album pics from last.fm but that is not your fault.

I agree. I've just started using this scraper instead of allmusic because of Last.fm's superior artist thumbs, but the thumbs scraped are dramatically compressed. Is this currently an unavoidable limitation or something simple I could improve on my end? Tweaking the thumbsize tag in advancedsettings is not making any difference. Hopefully I haven't resurrected this thread out of ignorance since my searches have got me nowhere.

Scraped

Image

Original (Last.fm)

Image
Reply

Logout Mark Read Team Forum Stats Members Help
Last.fm scraper in development - help wanted0