+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Development (/forumdisplay.php?fid=32)
+--- Forum: Scraper Development (/forumdisplay.php?fid=60)
+--- Thread: Last.fm scraper in development - help wanted (/showthread.php?tid=39028)
- TechLife - 2008-11-22 02:55
luv u spiff
- kriziz - 2008-12-07 11:26
hey guys, its good to find out somebody is working on more music scrapers. Most of my albums come up blank on allmusic.com. An alternative scraper to work around this issue would be great! Last.FM and discogs.com are my favorite sites, and i'd be happy to help out with scraper development for either site.
Is there any public place where i can find the current development version of your scraper ? Ofcourse i'd like to know where the development of the lastfm scraper is at. I have a lot of scripting experience (all sorts of stuff: PHP, Perl, pl/pgsql, LUA script, etc..) and have constructed a lot of complex regular expressions in the past, so if any help is needed, let me know..
- Aron Parsons - 2008-12-24 04:39
@pyro-x
Are you still working on this? Do you want any assistance? If so, post your latest revision and I'll see where I can help out.
- kastrolis - 2008-12-29 16:35
Pyro-x is either so busy with his work on the scraper that he doesn't even have time to check this message board, or he has given up on it altogether. if the second option were to be true, also taking in account that work on discogs.com scraper also seems to have been discontinued, there's no doubt - some other people should take over the alternative scraper development. personally I'm ready to start working on this scraper, however I have serious doubts that last.fm really is the best source of information, as the only real thing that it provides are album covers. discogs.com seems slightly better, but I don't like it listing all the different issues of the record (Canadian Vinyl editions and stuff like that).
- Aron Parsons - 2008-12-29 20:40
last.fm is sometimes good for more obscure artists and that's the main reason I want to scrape from it. Not all of the groups I listen to have entries at the more popular sites (e.g. AllMusic).
It's not currently possible to cascade scrapers, is it? For example, it can't find an artist with scraper 1, so it tries #2, then #3. Perhaps that is another bit of functionality that I can work on if others would find it useful as well.
- spiff - 2008-12-30 00:57
that would be neat, and some system to cross reference several scrapers in order to make matches more reliable
- kriziz - 2008-12-30 12:33
AllMusic.com isn't updated on a regular basis and doesn't list many relatively unknown artists. I'd expect a boost in search results from almost any new scraper, if its last.fm or discogs.. These two sites are both as good, provide a decent API etc.. One might be better than the other when it comes to specific musical nices. Most people are better off to use their favorite scraper for regular searches. We might want to make it a bit easier to switch scrapers, from the Album/Movie info screen or the manual search screen maybe ?
Anyway, in meantime I did some testing on a discogs.com scraper. This scraper script is just a few lines, it has CreateAlbumSearchUrl and GetAlbumSearchResults nodes and thats about it. Where I get stuck is testing the thing, XBMC simply crashes if I use it (no messages in xbmc.log).
The 'Scrap' test tool looks promising, however it is built to handle video scrapers, not music. Tried to fix that as well, but got caught up in C++ syntax
Scraper development would be much easier if anyone could fix that Scrap tool. Doesn't look like an easy job, though ..Am I missing a development tool here, should I be debugging from Visual C++ etc. ?
- Aron Parsons - 2008-12-30 17:24
kriziz Wrote:Where I get stuck is testing the thing, XBMC simply crashes if I use it (no messages in xbmc.log).
The 'Scrap' test tool looks promising, however it is built to handle video scrapers, not music. Tried to fix that as well, but got caught up in C++ syntax
Am I missing a development tool here, should I be debugging from Visual C++ etc. ?
I was fighting with the same thing last night. The 'Scrap' tool doesn't work correctly. Getting it to compile and run on Linux will be the first step, so maybe I'll try to get that sorted out over the upcoming weekend. Trying to debug scrapers through XBMC's GUI doesn't sound like fun.
- spiff - 2008-12-30 18:32
scrap is broken as we lost the source (blame donj for bad committing practices).
the way i do scrapers is using a regexp tool + printf. it works just fine for me, but an updated scrap tool would be invaluable for sure
- spiff - 2008-12-30 22:38
oh damn. kriziz, i have a semi-done discogs scraper (only album part)! we should try not to dupe work
if you want i'll gladly leave the artist part to you