[WIP] Audio-Matic Automated Music Downloader/Organizer

  Thread Rating:
  • 3 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #31
ProphetVX Wrote:I've attempted a similar project before, the biggest issue I found was the lack of sites to easily scrape accurately that actually had a comprehensive and accurate database, unfortunately with the exception of itunes, most sites have limited scope or catalog's, so the difficult part was merging that data across various sites in a useful manner. It seems like you've overcome those issues though, so keep up the good work, I'm very interested to see how this project progresses.

Between http://www.discogs.com and http://www.allmusic.com, most information can be obtained, but it is not always current. I cross-check the information obtained and validate it against any ID3 tag information that can be obtained from existing files. I haven't tried obtaining information from itunes, but I'm actually going to look into that right now since you mentioned it.

ProphetVX Wrote:Based on your screenshots, I'm assuming the application is written in C#? Another option is to port it to java, which there are many obvious syntactical similarities which would create a far easier path to having a hosted solution and platform independence, but that's obviously a fair way down the roadmap. Depending on the complexity of the project there is a plugin for netbeans that I've used with moderate success that can automate the porting of VB or C# to Java.

This project is currently written using VB.NET (2010 version). C# is very similar though... mostly minor syntax differences. I can fairly easily convert back and forth if needed. Converting VB.NET to Java would be a bit more of a challenge though. I'm at around 7,000 lines of code at the moment so that would definitely take some time.
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #32
Actually, I think I could write a scraper for itunes without too much difficulty (i'll have to play later)...

It looks like I can send a query on their website using something like this:

Code:
http://www.apple.com/search/?q=Breaking+Benjamin&section=ipoditunes

This may have some limitations though. I'll analyze it with firebug when I get some time.

Edit:

This is the ajax call structure call that happened when I searched for "Breaking Benjamin". It outputs a nice XML file for scraping...

Code:
http://www.apple.com/search/service/nph-search10?site=us_www&filter=1&snum=50&q=breaking+benjamin

Edit #2:

Even better! Looks like they have a documented API. Didn't think it would be that easy.

http://www.apple.com/itunesaffiliates/AP...rch2.1.pdf

And since I already scrape from AMG (AllMusic) IDs and iTunes allows you to do a search based on AMG IDs, this is easy as pie to add... but then again it looks like Apple may get their info from AllMusic so not sure how relevant this may be. If nothing else, I can at least pull out the iTunes album links (like this: http://itunes.apple.com/us/album/dear-agony/id332288664) and albumart Here's the sample query for the same artist though:

Code:
http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStoreServices.woa/wa/wsLookup?amgArtistId=534468&entity=album

What would be really nice is if I can extract information from existing iTunes Library XML files on a users PC if they use iTunes to manage their music collection.
(This post was last modified: 2011-03-16 05:29 by compcentral.)
find quote
DejaVu Offline
Posting Freak
Posts: 1,159
Joined: May 2010
Reputation: 2
Location: United Kingdom
Post: #33
Holy-shit-moly! This has come on a bundle since I first heard about the concept. Great work CC. I think the same as the rest though, should be web interfaced where possible, but we are all willing to wait - we have waited this long! A tad longer wont be too difficult!

I'm just a little concerned about the strain this could put on the NZB sites. Keep that in mind! Music will be sought after much more than TV & Movies and the gazillions of tracks available online will take a whole load of finding/indexing!

[Image: watched-fanart.jpg]
Maraschino - Web based HTPC Organiser. Keep track of all your web based programs all in one place.
Home Page - Repo - Bug Tracker - Support Forum.
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #34
DejaVu77 Wrote:I'm just a little concerned about the strain this could put on the NZB sites. Keep that in mind! Music will be sought after much more than TV & Movies and the gazillions of tracks available online will take a whole load of finding/indexing!

Thanks for the support. Interesting thought though. I will have a built-in rate limiter for search requests, but if enough people are using this that could still become a problem. Nothing else I can think of to help overcome that though other than falling back on the built-in torrent support.
find quote
bnevets27 Offline
Senior Member
Posts: 198
Joined: Feb 2009
Reputation: 1
Post: #35
With enough providers supported the load should be spread out pretty well. I'm sure Sick Beard has caused an increase in load too. But people have been using RSS feeds and other scripts before and still do, so I wouldn't expect it to be a problem.

Not knowing anything about coding, it might be best if you just continue to work in VB.NET. After you get the code cleaned up and working the way you like you could then release the code and the community could help translate it to python.

Can't wait to use this. No matter what platform its on. I can wait for linux support.
find quote
ProphetVX Offline
Member
Posts: 86
Joined: Oct 2010
Reputation: 0
Post: #36
DejaVu77 Wrote:Holy-shit-moly! This has come on a bundle since I first heard about the concept. Great work CC. I think the same as the rest though, should be web interfaced where possible, but we are all willing to wait - we have waited this long! A tad longer wont be too difficult!

I'm just a little concerned about the strain this could put on the NZB sites. Keep that in mind! Music will be sought after much more than TV & Movies and the gazillions of tracks available online will take a whole load of finding/indexing!

Initial searching would cause a spike in traffic but I highly doubt music would have more traffic than TV Shows. I certainly don't have a wishlist of 15-20 albums per week waiting to download. Nor would I do a back catalog of 200+ albums in one hit like you'd do with a TV Show. You can't download single tracks on newsbins, so unless you're an absolute audiophile who needs to download more than 2-3 albums a day you're unlikely to generate more traffic than videos would. In saying that all programs such as this should have rate limiting built-in. Scraping and having automated downloads is always a good excuse to get banned.
find quote
DazNoonan Offline
Senior Member
Posts: 257
Joined: Feb 2011
Reputation: 0
Post: #37
This looks fantastic, any plans for a Mac or Python based version?

Devices using XBMC
iMac | MacBook | MacBook Air | AppleTV2 | iPad | iPhone4
[Image: classic]
[Image: widget_1.png?1305454501]
find quote
rockstrongo Offline
Junior Member
Posts: 9
Joined: Feb 2011
Reputation: 0
Post: #38
Not sure if it's been mentioned, but I've used tagging software that utilizes the MusicBrainz database with great success. I'm not sure if they offer an API-like method of querying, but it might be worth a shot in looking to see if it's useful.

http://musicbrainz.org/

Edit: looks like they do indeed http://musicbrainz.org/doc/Developer_Resources
find quote
dannycorker Offline
Senior Member
Posts: 223
Joined: Mar 2010
Reputation: 0
Post: #39
Been waiting for something like this for ages! Cannot wait until release, any kind of timeframe?
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #40
rockstrongo Wrote:http://musicbrainz.org/doc/Developer_Resources

At one time I looked into this and planned to support it, but I'd forgotten all about it. I don't think there is a need for it in the initial release, but I will definitely add support for them in the future.

dannycorker Wrote:Been waiting for something like this for ages! Cannot wait until release, any kind of timeframe?

I wish I could give an exact date, but this is largely dependent on my work schedule and family life. I have two full-time jobs (own a computer business and work full-time as an IT Specialist for another company), a wife (need I say more), and 3 kids (4th is on the way... due any day now). That being said, I still usually find time to work on this at least a few times a week.
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #41
Ok.. I need help with part of this that I'm working on.

In the instance that the file naming conventions do not match the ID3 tag or .nfo file, which should be given the most weight? In other words, which do I assume to be correct in this circumstance? I could always query one of the online music DBs (allmusic, discog, etc), but what if there is no listing or conflicting information? I would alert the user, but should I make any assumptions here or just leave it out of the AMDB (Audio-Matic DataBase) until the issue is resolved?

I want to limit online lookups as much as possible (at least on the initial import) to keep it from getting too slow, but I also want to make sure the information is accurate.
(This post was last modified: 2011-03-16 23:16 by compcentral.)
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #42
Another little issue...

Right now, I have a rather advanced (at least I think so... spent hours on it) path scraper that will look for whatever naming pattern you define in the import process (see the 2nd set of pics on the 1st page of this thread). Literally any pattern and combination is supported and there is no limit on subfolder depth. Looks and sounds a lot easier than it was to code.

For instance, you could organize your music like this and you would be fine:
Code:
..\<genre>\<year>\<album>\<artist>\Song #<num>-<song>
Now hopefully no one is quite that insane, but it should work just fine for both importing and post-processing.

This is used for Artist/Album/Song detection if ID3 tag info is not available. The problem is, what do I do if some of the users files do not adhere to there own naming convention? Without tag info I really have nothing to fall back on. I guess I will just gather a list of all failed imports and instruct the user to manually import them. Sorry, I guess this turned into more of a vent than a question.

I guess the other issue would be if the user is too dumb to enter the correct naming pattern...

I suppose I could also examine a few of their folders and do an lookup online to determine whether the folders are artists, albums, genres, etc, but this would pretty much just be a guess. Do you think this is needed/wanted?
(This post was last modified: 2011-03-17 04:28 by compcentral.)
find quote
bnevets27 Offline
Senior Member
Posts: 198
Joined: Feb 2009
Reputation: 1
Post: #43
compcentral Wrote:In the instance that the file naming conventions do not match the ID3 tag or .nfo file, which should be given the most weight? In other words, which do I assume to be correct in this circumstance?

I assume the ID3 tag and .nfo files (maybe) are auto generated using a program. If that was true though the tag and nfo should match.... I guess you need to know which is more likely to be correct based on how albums are normally ripped.

compcentral Wrote:but I also want to make sure the information is accurate.
Glad your focused on this, I would prefer accuracy over anything else, its a pain to have to fix inaccuracies.


compcentral Wrote:The problem is, what do I do if some of the users files do not adhere to there own naming convention?

Do you think this is needed/wanted?

I wouldn't worry about it. I assume most people around here know that you should have your files organized properly for programs to work. I also think its not a bad idea to force user to follow a proper structure.

compcentral Wrote:I guess I will just gather a list of all failed imports and instruct the user to manually import them.

I think that's a fine solution. That way users can clean up their files when they import them.
(This post was last modified: 2011-03-17 08:49 by bnevets27.)
find quote
DejaVu Offline
Posting Freak
Posts: 1,159
Joined: May 2010
Reputation: 2
Location: United Kingdom
Post: #44
Hey compcentral,

Just a quick request (before the gun goes off!).
A few people I know use iTunes and are not to 'hot' on the procedures.

Are you planning to add a function that perhaps will add and automatically update iTunes (or other Music Library Managers) after a download has completed? Much like SB and CP can do with XBMC?

[Image: watched-fanart.jpg]
Maraschino - Web based HTPC Organiser. Keep track of all your web based programs all in one place.
Home Page - Repo - Bug Tracker - Support Forum.
find quote
dallasnights Offline
Fan
Posts: 490
Joined: Jan 2011
Reputation: 0
Post: #45
Second this request can not get itunes to work with SMB at all and really not sure how to get to my itunes on Windows 7 to my ATV with XBMC hope this programs solves that

Thanks
find quote
Post Reply