[WIP] Audio-Matic Automated Music Downloader/Organizer

  Thread Rating:
  • 3 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #46
DejaVu77 Wrote:Are you planning to add a function that perhaps will add and automatically update iTunes (or other Music Library Managers) after a download has completed? Much like SB and CP can do with XBMC?

iTunes uses proprietary encrypted .itl/.itdb files to store its music library. Updating this is not possible as far as I know. All I can do is read the .xml file that iTunes creates (which doesn't contain all info). Changes to this file do not effect the iTunes library, so nothing can be changed in the library. This is one of the many reasons why I do no use iTunes.

I plan on having support to update XBMC and I can add support for other music managers if needed provided there is some type of support for this.
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #47
I updated my 2nd post to reserve it for whatever I'm currently coding to save this from getting to long and to look for input from everyone as I code certain sections. If anyone has any thoughts on my import logic, now is the perfect time to post them.
find quote
DejaVu Offline
Posting Freak
Posts: 1,159
Joined: May 2010
Reputation: 3
Location: United Kingdom
Post: #48
This may get around the problem with auto updating iTunes...

http://itlu.ownz.ch/wordpress/

[Image: watched-fanart.jpg]
Maraschino - Web based HTPC Organiser. Keep track of all your web based programs all in one place.
Home Page - Repo - Bug Tracker - Support Forum.
find quote
jbalicki10 Offline
Member
Posts: 71
Joined: Mar 2009
Reputation: 0
Post: #49
Possible to add demonoid for the bit torrent sites?

Also do you need alpha.beta testers... Ill give it a shot...
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #50
DejaVu77 Wrote:This may get around the problem with auto updating iTunes...
http://itlu.ownz.ch/wordpress/

hasn't been updated since 2008... I kinda doubt it will work with iTunes 10+.

jbalicki10 Wrote:Possible to add demonoid for the bit torrent sites?

yeah.. I'm sure it could be an option. I will add others like isohunt as well once the core components are fully functional

jbalicki10 Wrote:Also do you need alpha.beta testers... Ill give it a shot...

I will definitely need testers, but I'm not quite ready for an alpha release yet. Getting closer every day though.
find quote
LoloMc² Offline
Senior Member
Posts: 163
Joined: Mar 2009
Reputation: 0
Post: #51
May I suggest that you add a clever way to do a replay gain on all files in the main folder ?
I explain : if the user has 150+ albums in the same folder, the replaygain process will work for hours. Would be great to get a background function which could be interrupted every time your software is closed.

Obviously, flac support is needed before...

FMI : is APE support planned ?


Regards,
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #52
LoloMc² Wrote:May I suggest that you add a clever way to do a replay gain on all files in the main folder ?
I explain : if the user has 150+ albums in the same folder, the replaygain process will work for hours. Would be great to get a background function which could be interrupted every time your software is closed.

Obviously, flac support is needed before...

FMI : is APE support planned ?


Regards,

replay gain tagging is supported in mp3 files as well, but I think that is (at least for now) outside the scope of this project. I'm not saying that it won't eventually be added, just not in the initial release. Thanks for the suggestion though. The main focus of this project right now is organization, download management, and new music discovery.

As for APE tags, does anyone really use them? I suppose i could add support for them if needed (provided I could find ample documentation on the format). Is ID3 tagging supported in flac files?
(This post was last modified: 2011-03-19 19:05 by compcentral.)
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #53
I updated the todo list and I now have the import function mostly done. just some more fine tuning to do yet.
find quote
HenryFord Offline
Posting Freak
Posts: 1,241
Joined: Jan 2010
Reputation: 19
Post: #54
It is like waiting for christmas when I was like 4 years old Smile

Really glad you get into this, I hope I can provide a little bit of support/help throughout the project - as soon as the code gets released I will take a look into it.

Thank You for developing this, I am really looking forward to it.
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #55
Glad to see some people are looking forward to this.

I've been able to spend a good deal of time refining the "import existing music" process over the last few days and it's really coming along nicely, but I have a few issues to sort out before moving forward... mainly how to handle naming conflicts.

Also, the artist/album/song detection logic is getting so complex that it literally took over an hour to import my library (roughly 25,000 songs, 1,000 artists, 4500 albums) so I need to see if I can optimize this. This is without any online lookups/scraping, which will be done optionally - either in the background as the app runs or initiated by the user - after the database is updated. I haven't tested the speed of this recently, but it will be largely dependent on internet connection speeds.

How does this speed compare to other music managers like iTunes or even XBMC? Can anyone comment on that? It's been a long time since I imported a large number of files into either one. I suppose I could give it a try... but it's probably a little premature yet. I'll definitely do a comparison once my code is closer to complete.

On the plus side, the import logic appears to be very accurate (more so than iTunes in my tests) and can detect a number of naming/tag-related issues. The problem is deciding what to do after detecting these problems. Since there isn't always a definitive way to determine what is correct and what isn't, I use all of the available information to make a "best guess" at the correct information. This ends up almost always being accurate, but I don't want to assume that my "best guess" will always be correct in case a mistake is made.
find quote
vike105 Offline
Junior Member
Posts: 35
Joined: Apr 2009
Reputation: 0
Post: #56
Must......Have......This......Now!!!.....lol...great job, keep it up
find quote
ProphetVX Offline
Member
Posts: 86
Joined: Oct 2010
Reputation: 0
Post: #57
compcentral Wrote:Glad to see some people are looking forward to this.

I've been able to spend a good deal of time refining the "import existing music" process over the last few days and it's really coming along nicely, but I have a few issues to sort out before moving forward... mainly how to handle naming conflicts.

Also, the artist/album/song detection logic is getting so complex that it literally took over an hour to import my library (roughly 25,000 songs, 1,000 artists, 4500 albums) so I need to see if I can optimize this. This is without any online lookups/scraping, which will be done optionally - either in the background as the app runs or initiated by the user - after the database is updated. I haven't tested the speed of this recently, but it will be largely dependent on internet connection speeds.

How does this speed compare to other music managers like iTunes or even XBMC? Can anyone comment on that? It's been a long time since I imported a large number of files into either one. I suppose I could give it a try... but it's probably a little premature yet. I'll definitely do a comparison once my code is closer to complete.

On the plus side, the import logic appears to be very accurate (more so than iTunes in my tests) and can detect a number of naming/tag-related issues. The problem is deciding what to do after detecting these problems. Since there isn't always a definitive way to determine what is correct and what isn't, I use all of the available information to make a "best guess" at the correct information. This ends up almost always being accurate, but I don't want to assume that my "best guess" will always be correct in case a mistake is made.

Simple question, but when you are scanning and processing the information.. Is your process multi-threaded? With VB.NET it's quite common for developers to not take advantage of threading, it'll make a huge difference when dealing with large datasets. I assume given the complexity of the project you've already taken this route, but even just 1 thread handling the file I/O and then another thread processing a queue would increase performance exponentially.

Another way to speed up the processing is to make sure your regex is compiled and not interpreted every time it goes through a loop etc. Your application can take a severe performance hit if the interpreter needs to translate the regex pattern every time it uses it.

In saying that for a large collection taking an hour in the initial import isn't necessarily slow. The biggest cost is the I/O, and unfortunately much of the time there is little you can do to optimise it.
(This post was last modified: 2011-03-23 22:16 by ProphetVX.)
find quote
LoloMc² Offline
Senior Member
Posts: 163
Joined: Mar 2009
Reputation: 0
Post: #58
Quote:NEED HELP: Give me ideas!

* Need to find a way to handle non-standard characters in names/tags (ie. Sérgio Mendes, Mötley Crüe, Queensrÿche). The biggest problem is determining the correct spelling. Should I just convert ö to o, é to e, etc.?
Yes, do the simplest way. This is not a big problem Wink

Quote: * What should be done when conflicting information such as artist or album names are detected (ie. "The Beatles" for some files and "Beatles" for others)? Which one is right and by what authority? I have some ideas on this already, but I'd like to hear some others

I'd go for an editable list of words. I explian : "The"/"Der"/"Le","La"/"El" are concerned (and much more languages but I just speak English and some words of German/French and Spanish)

Just my 2 cts
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #59
Thanks for your input. It is greatly appreciated.

ProphetVX Wrote:Simple question, but when you are scanning and processing the information.. Is your process multi-threaded? With VB.NET it's quite common for developers to not take advantage of threading, it'll make a huge difference when dealing with large datasets. I assume given the complexity of the project you've already taken this route, but even just 1 thread handling the file I/O and then another thread processing a queue would increase performance exponentially.

You are correct. Currently I'm not doing any threading. Here is the basic process that accounts for most of the import function. I admit my experience creating multi-threaded apps is limited. How would you recommend threading this?

Once I find an Artist path, I generate a list of all music files within that folder and all subfolders. I then scan each of these files for tagging metadata and write everything discovered for the file to the database. After all files have been scanned, I examine the combined tagging info to look for inconsistencies. Then I write the information obtained for this artist into the artist table and if naming conflicts are found, write that information to the database as well before moving on to the next artist folder and repeating the process.

ProphetVX Wrote:Another way to speed up the processing is to make sure your regex is compiled and not interpreted every time it goes through a loop etc. Your application can take a severe performance hit if the interpreter needs to translate the regex pattern every time it uses it.

Rather than simply use regex, I wrote a custom function to parse the filename and compare it to the folder structure definition. It seemed to nearly as fast and offered more flexibly for inexact matches (items that did not conform to the folder structure pattern defined before the import). I'll have to take another look at this and see how much of a performance hit/gain I get revising it. Up until this point, my main concern was functionality, but soon enough optimization will take a more important role.

ProphetVX Wrote:In saying that for a large collection taking an hour in the initial import isn't necessarily slow. The biggest cost is the I/O, and unfortunately much of the time there is little you can do to optimise it.

Yeah. I agree. I think all the disk I/O is causing most of the slow down.
(This post was last modified: 2011-03-24 00:10 by compcentral.)
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #60
LoloMc² Wrote:I'd go for an editable list of words. I explian : "The"/"Der"/"Le","La"/"El" are concerned (and much more languages but I just speak English and some words of German/French and Spanish)

good suggestion, but the issue is not just saying "the beatles" = "beatles", but determining which name is the "right" one to use when post-processing and renaming downloaded files. I actually already ignore things like "the, &, and, etc." when searching for downloads and doing scraping and API calls, but that's another sort of issue. I like to have my music named "perfectly" if possible... hence this application was born.
find quote
Post Reply