[WIP] Audio-Matic Automated Music Downloader/Organizer

  Thread Rating:
  • 3 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #1
[Image: title.png]

Website: http://www.audio-matic.com

Twitter: https://twitter.com/#!/Audio_Matic

NOW IN ALPHA TESTING PHASE!

I've been working on this project off an on for several months and now I finally have a semi-working version ready for testing. During the Alpha testing stage, I will be releasing bug-fix releases quite often so please check back often and make sure you have the latest version installed before posting a bug. All downloads will be posted on http://www.audio-matic.com.

Alpha Testing Registrations are now open again at http://www.audio-matic.com!

The goals for this project are as stated on the website:
  • Automatically download new album releases by your favorite artists.
  • Easily manage your music collection and remove duplicate songs.
  • Discover new music that's similar to what you already know and love.
  • Sample new releases and popular music before you download it.
  • Automatically sort your files and fill in missing ID3 tag info.

Here are some of the components that are working or are mostly complete:
  • Option to replace upper ascii characters with lower ascii equivalents (ie. ñ = n, Ö = O, etc.)
  • XBMC notifications and updates are both working
  • Complete Discogs.com API Interface and XML reader
  • Fully functional allmusic.com scraper
  • Complete iTunes API interface and JSON reader
  • Automated image downloading for covers, artists and fanart
  • Tag reader supporting all popular music file types (.mp3, .wma, .m4a, .m4p, .flac, .aac, .ogg, .ape, .wv, .wav, .aif, .mp1, .mp2, .mpc, .mp+, .mpp, and .mka) using taglib-sharp library.
  • SQLite Database (mySQL was working as well, but was abandoned to simplify installation)
  • Fully automated database creation (no database setup/installation required)
  • NZBMatrix API to search for album .nzb's
  • SABNzbd+ API Interface to automate downloading
  • Torrent support using thePirateBay (others will follow)
  • Builtin Media Player
  • Artist/Album/Song detection using File/Folder names and/or Tags and verification using allmusic/discogs
  • Automated Post-Proccessing for renaming and audio tag updates
  • Same-name artist detection and resolution (asks user to select the correct artist... could probably be improved)
  • Discover new music using - AllMusic Editor's Choice, AllMusic New Releases, NZBMatrix New Releases, BillBoard Charts (needs finished), iTunes popular albums (will be added with iTunes scraping)
  • Support for automatic tagged music downloading... currently just Shazam tagged tracks, but will expand this later. So if you here a song you like, using the Shazam iPhone app, let Shazam detect the song name and the corresponding album will be downloaded automatically. I would like to add support for SoundHound and others as well.
  • Problem file & folder detection
  • Duplicate song detection
  • Song preview using AllMusic and iTunes streams

To Do / Working on:
  • Allow 'Wanted' downloads to be queued.
  • Add UseNet retention limit to settings.
  • Export Library to text/csv/html.
  • Display Library stats (total playtime, total artists/albums/tracks, etc.).
  • Add support for CD folders (ie. CD1, CD2, etc.).
  • Add support for Torrent/NZB RSS feeds.
  • Add options for preferred sound quailty to download (lossless/lossy, min bitrate, file format, etc.) Try to use scene release types to help determine this.
  • Add support for MusicBrainz.com, last.fm, Amazon, and others.
  • Use similar artists, genre, and mood info scraped from allmusic/others to suggest new downloads.
  • Complete Billboard Chart feed feature.
  • Finish http://www.albumartexchange.com scraper.
  • Add option to download all fanart from site like htbackdrops.com or Wallbase.cc into folder Extrafanart for each artist.
  • Custom playlist creation for XBMC.
  • Create a web user interface/API.
  • XBMC .nfo file generation.
  • Allow songs to be marked as archived.
  • Add 'Edit Artist' feature to allow you to modify the provider info of all artists. Mainly useful if the wrong provider ID is associated with an artist.
  • Add support for music videos.
  • View artists by genre.
  • Allow users to rate artists, albums, tracks and use this to create playlists (need to save ratings to database).
  • Speed up New Releases load time and verify it is still operational.
  • Add option to hide new release albums that you already have or mark them as already downloaded.
  • Create Consolidated Album view and show local albums not recognized in provider listings.
  • Verify min/max size for search results is working.
  • Create a way to handle Various Artists, Compilations, Soundtracks, Unknown Artists, etc.
  • Add default sorting methods for Album/Song listviews to Settings.
  • Allow user to ignore/hide provider albums (possible duplicates, etc.)
  • Create a Band Members view using information from allmusic.com to find all bands an individual has been a member of.
  • Add option to use post-processing on previously downloaded files.
  • Explore adding an integrated cue sheet splitter for lossless files if the copies are made with EAC.
  • When adding a folder source, exclude read-only folders, system folders, etc.
  • Create an 'exempted' folder to exclude from imports.

Alpha 0.27 Release Notes
  • BUG FIX - Fixed the 'ShowUI' exception when attempting to delete an existing album folder.
  • FEATURE - You can now remove a single artist from the Manage Artist screen.
  • REVISION - The last used Import Folder Structure is now remembered.
  • REVISION - Import errors are now suspressed until the import is complete.

Known Issues / Reported Bugs:
  • Folder.jpg images will be overwritten without warning (not really a bug, but will add an option to alert user before overwriting).
  • Adding multiple source folders is pretty much untested and currently each artist can only have one root folder otherwise something may crash.
  • Duplicate song details are not yet shown even though they are detected.
  • For now, the individual File Tag view (shown in one of the screenshots) has been disabled. I made some recent code changes that require that I need to recode this but have not yet had the time.
  • Application can become unresponsive when importing/post-processing (should be fixed when multi-threading is in place).
  • Sometimes an invalid artist id occurs when importing but you can manually add the artist afterwards.
  • upper/lower case problem storing provider id's in database?
  • iTunes Provider ID link does not currently link to anything.
  • Blank username and password in NZBmatrix download.php API string causing problems for some users?
  • 'Problem Folder' Tool does not take folder structure into consideration (old code... needs updated).
  • KERNELBASE.dll Exception (code: 0xe0434352) occurring on startup if Windows Media Player is not installed.
  • 'Update Artist' button doesn't force a discography provider update correctly (needs to remove old info) and scrape artist info.
  • Artist image does not always load until after the program is restarted.
  • Sometimes users recieve a 'folder structure does not match' error when they should not.
  • Importing of artists like 'AC/DC' causes the artist to be imported incorrectly as 'AC'.
  • Sometimes album status icons still don't update until the album is selected for some reason.
  • Track durations are not always calculated correctly for some reason.
  • All band members are not showing for discogs listings.
  • Need to add more error handling for 'Argument 'Start' must be greater than Zero' errors.
  • Post-processing may not work correctly if using UNC or relative paths.
  • If you add a folder source with no music in it, it crashes?
  • If you set an invalid music folder in settings and click on the post processing tab it crashes?
  • If Genre is in path, it should be determined on an 'artist' level.
  • Need to validate renaming structure (ie. <num> can't exist at the artist/album level of path).
  • Need to update library (scan for changes) and remove old data when data is updated (partially in place).
  • Finish Newzbin NZB provider code.
  • Problem with apostrophe in album name and recognizing... 12 stones - Potter's Field.
(This post was last modified: 2011-07-17 17:38 by compcentral.)
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #2
Screen Shots:

[Image: main.jpg]

[Image: manage%20artists.jpg]

[Image: new%20releases%20-%20allmusic%20editors%20choice.jpg]

[Image: new%20releases%20-%20allmusic%20weekly.jpg]

[Image: new%20releases%20-%20nzb.jpg]

[Image: id3%20tag%20viewer.jpg]
(This post was last modified: 2011-05-23 22:48 by compcentral.)
find quote
butchabay Offline
Skilled Skinner
Posts: 3,782
Joined: Mar 2010
Reputation: 74
Location: Planet Crematoria
Post: #3
For sure i'm going giving a try, thanx!
find quote
compcentral Offline
Fan
Posts: 626
Joined: Apr 2010
Reputation: 2
Post: #4
    [Revised Often] PROGRESS - CURRENTLY CODING:

    This space is reserved for updates on whatever I'm currently working on and how it is progressing. Sometimes I could use help from potential users like YOU. I appreciate any feedback you can give me.

    [COMPLETE] Improved Artist/Discography Matching:

    Now when you add an artist manually (not using the add source/import), you will be asked to select the artist listing that matches on each of the discography providers.

    [Image: ArtistMatches.png]

    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


    [COMPLETE] Duplicate Song Detection & Removal:
    I've spent the last two days working on this one. I now have a way to generate a reliable list of duplicate songs. This is how I detect duplicates:
    • When files are added to Audio-Matic's library, a "hash" is created using the trackname and stored in the database. I'll explain how this hash is created in a moment.
    • The database is then queried to find records with duplicate hashes.
    • If songs with the same hash value have the same artist and the difference in duration of the two songs is less than x number of seconds (i use 15 in my tests), they are considered to be possible duplicates.
    • Since the same song could appear on compilation albums, soundtracks, etc., I will add the option to only consider a song duplicate if the album hashes match as well.

    The hash is created using this algorithym:
    1. Convert the song name to lower case.
    2. Remove noise words (ie. a, an, the, and, etc.).
    3. Convert 2-digit numbers to corresponding words (ie. one = one, 30 = thirty, etc.).
    4. Convert upper ascii characters into alphabetic characters (ie. ñ = n, Ö = O, etc.).
    5. Strip away all remaining non-alphanumeric characters including spaces.


    I also have code in place to select either the highest or lowest quality version of each song. I may add a few more options and maybe some additional file information, but here is what I have so far:

    [Image: duplicates.png]

    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

    [COMPLETE] Fine Tuning the File Import Process:
    This is the new "Add Source" concept I'm currently working on for importing your music into Audio-Matic.

    [Image: music%20source.jpg]

    After making a selection, you can fine tune the suggested folder naming structure:

    [Image: naming.jpg]

    A similar format will be used for both more accurate/flexible importing and for post-processing (renaming). As you edit the naming structure the sample file path/name is changed and it is validated before adding the source folder to the Audio-Matic library.

    While I wanted to make the folder naming structure in such a way that any possible naming convention could be used, it looks like it would make sense to put some limitations on this to help make artist/album/song detection faster and more reliable (especially if id3 tags are not available).

    Here are the only file/folder naming restrictions:
    • The <artist> tag must come before the <album> and/or <song> tag.
    • <album> tags are optional (but recommended), but if used, they must appear immediately after the <artist> tag.
    • Currently, using a space as a separator can be an issue (ie. ..\<artist>\<album> <year>\... since spaces could also occur in the album name). I'm working on a better way to handle this though.

    Examples:
    Code:
    ..\<genre>\<artist>\<album>\<artist> - <song>
    Code:
    ..\<artist>\<album> - <song>

    IMPORT SCRIPT LOGIC - Artist Detection (A work in progress):

    Here is how the import function works to make artist/album/song detection as accurate as possible:

    1. [DONE]Use folder naming structure to determine the folder level where artists are located.
    2. [DONE]Generate a list of all music files found in each Artist folder (and it's subfolders).
    3. [NO LONGER SUPPORTED]If no artist folder is defined (ie. artists are only defined as part of the filename), handle this differently.
    4. [DONE]Gather as much ID3 tagging information as possible from each Artist's files, but most importantly, the artist, album, year, and genre.
    5. [DONE]Since some artists may have many name variations (ie. 30 Seconds to Mars, Thirty Seconds to Mars), log all artist name variations found in tag info as well as the name of the folders/files.
    6. [DONE]Since there may be multiple artists with the same name (ie. there are at least 7 bands with the name: "Red"), album names and genre's will be gathered in the same manner to help make a guess as to the correct artist. Song titles and the years of album releases are also be used for this purpose.
    7. [DONE]If naming conflicts are discovered, information from online sources (several if needed) is gathered and compared to the information obtained from tags/filenames to determine the most likely artist.
    8. [DONE]Inconsistencies in naming and tag info are logged and can be later corrected once the user has confirmed that the "most likely" artist is correct.
    (This post was last modified: 2011-05-23 22:49 by compcentral.)
    find quote
    NTMID8 Offline
    Junior Member
    Posts: 15
    Joined: Jan 2011
    Reputation: 0
    Post: #5
    Support for NZB's R Us (nzbsrus.com) please
    find quote
    dallasnights Offline
    Fan
    Posts: 487
    Joined: Jan 2011
    Reputation: 0
    Post: #6
    WOW nice, maybe you can make it work to download the Itunes stuff one has on thier PC to ATV XBMC, I seem to always have a problem getting my music and video to work with SMB
    Thanks and keep up the great work
    find quote
    compcentral Offline
    Fan
    Posts: 626
    Joined: Apr 2010
    Reputation: 2
    Post: #7
    NTMID8 Wrote:Support for NZB's R Us (nzbsrus.com) please

    Support for all major NZB providers will be added once the core functionallity is complete. Additional torrent providers will also be added as well.

    I just have NZBmatrix for now to keep things simple.

    dallasnights Wrote:WOW nice, maybe you can make it work to download the Itunes stuff one has on thier PC to ATV XBMC, I seem to always have a problem getting my music and video to work with SMB
    Thanks and keep up the great work

    Thanks for the encouragement. I will be looking into ways to better work with iTunes, but iTunes is evil when it comes to file management. This will probably be one of the last items on my todo list since I personally hate iTunes (sorry apple people) but I will still devote some time to this later on. Thanks for the suggestion.
    (This post was last modified: 2011-03-14 01:52 by compcentral.)
    find quote
    jbalicki10 Offline
    Member
    Posts: 71
    Joined: Mar 2009
    Reputation: 0
    Post: #8
    One of the major pet peeves of mine of ANY music manager is that they dont support a folder.jpg for the Artist/Band. (IE a group pic)

    Example:

    Genre/BandorArtist/Album1
    Genre/BandorArtist/Album2

    c:\Music\Rock\PearlJam
    |-folder.jpg (Band Pic)
    c:\Music\Rock\PearlJam\Ten
    |-Album cover art (folder.jpg)
    c:\Music\Rock\PearlJam\VS
    |-Album cover art (folder.jpg)
    etc....

    Might be Nice to have a search that finds all the Bands that need pics...
    find quote
    compcentral Offline
    Fan
    Posts: 626
    Joined: Apr 2010
    Reputation: 2
    Post: #9
    jbalicki10 Wrote:One of the major pet peeves of mine of ANY music manager is that they dont support a folder.jpg for the Artist/Band. (IE a group pic)

    Example:

    Genre/BandorArtist/Album1
    Genre/BandorArtist/Album2

    c:\Music\Rock\PearlJam
    |-folder.jpg (Band Pic)
    c:\Music\Rock\PearlJam\Ten
    |-Album cover art (folder.jpg)
    c:\Music\Rock\PearlJam\VS
    |-Album cover art (folder.jpg)
    etc....

    Might be Nice to have a search that finds all the Bands that need pics...

    This has already been implemented.
    find quote
    Cuddly Cannibal Offline
    Junior Member
    Posts: 30
    Joined: Feb 2011
    Reputation: 0
    Post: #10
    Wow, this looks great! I've been waiting for something like this to pop up, kudos to you sir for making something with some awesome potential.
    find quote
    compcentral Offline
    Fan
    Posts: 626
    Joined: Apr 2010
    Reputation: 2
    Post: #11
    Thanks, I've put a ton of time into this. I can't wait to have a final product to share. I've been just working on it in my spare time (which I don't always have much of) but it's come a long way so far. I wish I didn't have two other full-time jobs so I could spend more time on this.
    find quote
    Aenima99x Offline
    Donor
    Posts: 1,128
    Joined: Aug 2007
    Reputation: 8
    Location: California
    Post: #12
    So it's only Windows? Why not follow SickBeard & CouchPotato and make it cross-platform with a web interface?
    find quote
    ptipton Offline
    Senior Member
    Posts: 128
    Joined: Feb 2008
    Reputation: 0
    Post: #13
    Couple of questions:
    Will it be cross platform? A linux version would be great.
    Will there be the option to choose quality, e.g. flac, mp3 etc.

    Looks like a great tool and am looking forward to trying it.

    Thanks
    find quote
    bnevets27 Offline
    Senior Member
    Posts: 198
    Joined: Feb 2009
    Reputation: 1
    Post: #14
    Honestly haven't even finished reading the OP but I just thinking today how much the need for something like this is.

    EDIT: Read everything now. I agree with the above two posts. I really hope that is will run on linux with a web interface. I have all the same programs that you have in your signature running on an unraid server (linux) and would really hate to have to have a windows machine running all the time for just this program.

    I really look forward to using this and experience its development.

    I know this is a little early to be requesting a feature but I figured I would write it down and see if it sounds useful to anyone else.

    I would love if Audio-Matic was able to take a simple list of songs. Find the albums that those songs are on, download the albums then create a playlist that matches the simple list of songs.

    The purpose of this is that Audio-Matic could take a list (that I provide it) of recently played songs off a radio station I like and it could match that list. I've been trying to find a way to do this and would be a great to see it in Audio-Matic.

    One other idea, just putting it out there, is the ability to record internet radio streams. Having the ability to record a radio show would be nice. Basically record a stream on a schedule. I know this is a whole other element, just food for thought.

    Thanks for stepping up to create this.
    (This post was last modified: 2011-03-14 06:30 by bnevets27.)
    find quote
    davewantsmoore Offline
    Junior Member
    Posts: 23
    Joined: Nov 2008
    Reputation: 0
    Post: #15
    Aenima99x Wrote:So it's only Windows? Why not follow SickBeard & CouchPotato and make it cross-platform with a web interface?

    Please do this. You will attract SO many more users this way.
    find quote
    Post Reply