MusicBrainz & extending the MusicScanner/Scraper

Will definitely give this a try

Great :-)

Two things I noticed while doing this:
1) The AllMusic scraper is broken (ergh)
2) The Universal Scraper can query MusicBrainz too fast... When I scan my library MBs web service starts rejecting my queries as they seem to have a rate-limit at 22queries / second and reject anything above that. I think we might need to look into implementing a rate-limit per-domain on queries...? Is there already one?

I've been looking at the GUI side of this for a few days trying to understand the best direction to go with this and I had a few questions..
Wanted to ask a few questions:

The musicdb:/3/1 style URI schema.. is that coming or going? I notice music FileItems use their real path but then later we convert that to a musicdb://3/1 style URI just to pass through the ID to the schema and get the scraper for the path. It seems like in the Video DB, either the files have the videodb:// style schema in the first place or we use a DB call to get it from the file path?

The VideoDB GUI/database seems to work almost entirely from FileItems, rather than the music code which is a mix of paths, strings, Album and Artist type structs and FileItems. E.g. for GUIWindowMusicBase we take an Album or Artist from the DB and convert it to a FileItem for listing... In GUIWindowMusicBase we then convert it back to an Album or Artist, then just extract the strings from it for FindAlbumInfo and find the album and then pass an Album to MusicInfoDialog.... I think it would make sense to try and tidy some of this up for consistency and use one method throughout.. That could be FileItem, or I could propagate Albums/Artists through or MusicInfoTag... Then there is also MusicAlbumInfo which is propagated to the GUI code in some places I think unnecessarily.. I feel this just needs to live in DownloadAlbumInfo for showing the Selection dialog.

Wanted to get a few ideas around this... it won't change the functionality as I think I can make this work with any of those schemas, it's just to understand which way people envision this working.

Also what do the SetPropertiesForFileItem, SetPropertiesFromArtist etc calls do? There seem to be text based properties on FileItems, but I can't find where these are actually read or used. Can you point me to a reference please?

They're read directly by the skin - the former in lists, the latter most likely in the artist info dialog.

OK, so I patched my system, and removed my library to test. I rescanned a single directory with this release:

http://musicbrainz.org/release/c7c5e6b3-...92595c8c19

It's a release with 2 artists: "Technical Itch" and "Kemal", I expected that now with using the MBIDs that I would see two separate artists, with the same release listed underneath them, but I still see one single artist. I guess that part isn't implemented yet. (This is the killer feature for me, as well as disambiguating identically named artists).

I should mention that I've already tagged the files with picard and they already contained MBIDs etc. Here's the contents of the db:

Yes, the MusicBrainz IDs/disambiguation are not done yet. I need to clean up the GUI code as there are a bunch of areas we rely on artist/album name that will break if we add duplicate artists right now. But I wanted to get some direction from the XBMC team (See 3 posts up), and also I'm semi waiting for the first round of patches to go in first and get the feedback from that cycle so I don't repeat any mistakes and to manage the workload/support concerns. This PR contains all of the required database changes though so the support concerns for the next set of patches should be significantly smaller.

Given the feedback so far I suspect there are maybe 3-4 more PRs to get this done:
1) GUI code cleanup
2) Any other disambigutation/uses of GetArtistByName etc.
3) MusicBrainz into the DB
4) Scraper code

If I don't get any direct feedback in the next day or so I'll just go ahead and put a PR with it done my own (using CAlbum/CArtist).

As a side note while I'm writing, in the latest upstream/master the Universal Music Scraper is broken for some of my artists (stack overflow due to recursive calls in Run/RunNoThrow in ParseMBDiscography). This happens with a fresh music DB on a clean upstream/master (I had to go check as I figured it was my code). I need to go file a report on the Universal Music Scraper threads but the bug looks to be in core.

About the stack overflow (i think):
http://forum.xbmc.org/showthread.php?tid...pid1129456

Core code isn't my expertise but sure hope some one will look at this.

Thanks Martijn, thats the one I saw. It seems he's fixed the scraper, but I think it really tickled a bug in core. I spent 2-3 hours with this up in the debugger last night but I'm not aware enough of the deep internal scraper code yet to find a fix myself. And I hate XML parsing code :-)

Did you also link the scrapers to the database or is that something that needs some more work?