[RELEASE] Archive.org (Music Plugin) Live Music Archive

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Thumbs Up  [RELEASE] Archive.org (Music Plugin) Live Music Archive
Post: #1
Requested by waverz . (Sorry it took me so long, I had it 90% done like two weeks ago then forgot about it until I was cleaning out some files on my computer.)

download from http://code.google.com/p/rwparris2-xbmc-...loads/list

Note that some bands listed with only one entry actually only have 'collections' (whatever that is) so when you click on them you'll get a blank directory. I may add some sort of error handling in the future to notify you why nothing is there, but at the moment that isn't likely.

Also note that some shows don't have an XML file associated with them. I don't know why, but those won't work because I don't have a good way to pull out the info.

Aside from that, if you notice any bugs and/or have feature requests let me know in this thread and I'll get them fixed up.
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #2
http://code.google.com/p/rwparris2-x...downloads/list

Now with less-stupid sorting!
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #3
Is this working for anyone? 20 downloads and zero comments make me a sad panda.
find quote
waverz Online
Member
Posts: 62
Joined: Jul 2007
Reputation: 0
Post: #4
Cool dude, i thought you forgot about me, i have it working here!! Not the prettiest thing in the world but it does what i wanted it to do for sure.

Thanks a ton man!!
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #5
Any suggestions on how to make it "prettier" ? Aside from a bit of minimal testing, I've only used the plugin for two bands so there may be some stuff that I didn't notice.

I'm pretty new to plugins so ALL feedback is welcome.
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #6
Archive.org changed their site. Updated plugin to work with them (and fixed that stupid <span class=searchTerm> thing)

When using this, keep in mind that archive.org will sometimes just not serve up files. So if you click on something and then it doesn't load anything and kicks you out of the directory, just try again.

And if a file doesn't work with mplayer, try paplayer (context>play with...>paplayer). It seemed to work better on than the default on some files.

download from the site in my sig or use the xbmc addons installer.

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
find quote
ChinaRider Offline
Junior Member
Posts: 15
Joined: Oct 2008
Reputation: 0
Post: #7
I downloaded the live music archive plugin and installed to my xbox xbmc.

When I go to the music plugins menu, I see the plugin and clicking on it gives me the ability to drill down to the artist / band level. However, every artist that I've tried clicking on results in a screen with nothing to choose from, only a '..' up arrow to return to the previous menu.

I've even verified that the artists I'm choosing have actual performances in the archive online. For example, I chose the band '2me' and they have three performances available for streaming or download. But on the xbmc plugin, it just shows me an empty directory.

What am I missing here?
find quote
flacnvinyl Offline
Member
Posts: 53
Joined: Nov 2008
Reputation: 2
Post: #8
Same problem here.
find quote
tylerh Offline
Junior Member
Posts: 2
Joined: Jun 2009
Reputation: 0
Post: #9
I had the same problem. I tweaked the default.py script slightly and got it to work. Thanks rwparris for building the plugin!!!

Here's the fix I made:

1: Open the default.py file (from the Live Music Archive folder) in your text editor
2: Change line 56 to the following (one Tab in front):
Code:
p=re.compile('<li><a href="(.+?)">(.+?)</a>.+?<a href="(.+?)">(.+?)</a>')
3: Change line 79 to the following (two Tabs in front):
Code:
addDir(bandList[i][1]+'  ('+bandList[i][3]+')',bandList[i][2],4)
[EDITED TO ADD:]
4: Change line 88 to the following (two Tabs in front):
Code:
p=re.compile('href="(.+?)">.*(Live at.+?)</a>')

Now your band list should include the # of shows that are available, and they should work when you select them.

If you prefer not to see the # of shows, change line 79 to the following instead of the code listed in step 3 above (still two Tabs in front):
Code:
addDir(bandList[i][1],bandList[i][2],4)


It seems the necessary link moved from the band's name to the (# shows) link. I just had to tweak the regular expression, and the handling to match. Also, fixed the line in Step 4 which was breaking some show lists.


Remaining issue: The plug-in works for me now with all the individual file types, but not with the m3u playlists. XBMC says there's an error, but no details, and I'm TOTALLY new to python and xbmc plugins - although I did play around with it a little with no success.

Maybe someone else (rwparris, maybe?) has ideas as to why it chokes on building links for the m3u playlists?

Here is the full default.py file, in case you have difficulty finding the specific lines and want to just replace the whole thing:
[INCLUDES LATER EDIT FOR SHOW LIST CORRECTION]
Code:
#Live Music Archive (archive.org) default.py
#by rwparris2

import urllib,urllib2,re
import xbmcplugin,xbmcgui
from BeautifulSoup import BeautifulStoneSoup, BeautifulSoup

xbmcplugin.setContent(int(sys.argv[1]), 'songs')
xbmcplugin.addSortMethod(int(sys.argv[1]), 7)
xbmcplugin.addSortMethod(int(sys.argv[1]), 9)


def getLink(site):
    req = urllib2.Request(site)
    req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
    response = urllib2.urlopen(req)
    link=response.read()
    response.close()
    return link

def addDir(name,url,mode):
    u=sys.argv[0]+"?url="+urllib.quote_plus(url)+"&mode="+str(mode)+"&name="+urllib.quote_plus(name)
    ok=True
    liz=xbmcgui.ListItem(name)
    liz.setInfo( type="Music", infoLabels={ "Title": name } )
    ok=xbmcplugin.addDirectoryItem(handle=int(sys.argv[1]),url=u,listitem=liz,isFolder=True)
    return ok

def addLink(name,url,artist,album,track):
    ok=True
    liz=xbmcgui.ListItem(name)
    liz.setInfo( type="Music", infoLabels={"Title": name,"Album":album,"Artist":artist, "Track":track } )
    ok=xbmcplugin.addDirectoryItem(handle=int(sys.argv[1]),url=url,listitem=liz)
    return ok

def get_params():
    param=[]
    paramstring=sys.argv[2]
    if len(paramstring)>=2:
        params=sys.argv[2]
        cleanedparams=params.replace('?','')
        if (params[len(params)-1]=='/'):
            params=params[0:len(params)-2]
        pairsofparams=cleanedparams.split('&')
        param={}
        for i in range(len(pairsofparams)):
            splitparams={}
            splitparams=pairsofparams[i].split('=')
            if (len(splitparams))==2:
                param[splitparams[0]]=splitparams[1]
    return param

######################################################################
def getBandList(link):
    bandList=[]
    p=re.compile('<li><a href="(.+?)">(.+?)</a>.+?<a href="(.+?)">(.+?)</a>')
    match=p.findall(link)
    for band in match:
        bandList.append(band)
    return bandList

def alphList(bandList,letter):
    alphList=[]
    if letter=='0':
        num='.0123456789'
        for n in range(0,10):
            for i in range(len(bandList)):
                if bandList[i][1][0]==num[n]:
                    alphList.append(bandList[i])
    else:
        for i in range(len(bandList)):
            if bandList[i][1][0]==letter:
                alphList.append(bandList[i])
    return alphList

def showBandList(bandList):
    i=0
    while(i<len(bandList)):
        addDir(bandList[i][1]+'  ('+bandList[i][3]+')',bandList[i][2],4)
        i+=1
    return bandList

def getShowList(link):
    tree=BeautifulSoup(link)
    d=tree.findAll('a', 'titleLink')
    showList=[]
    for i in range(len(d)):
        p=re.compile('href="(.+?)">.*(Live at.+?)</a>')
        match=p.findall(str(d[i]))
        for show in match:
            showList.append(show)
    for n in range(len(showList)):
        s=re.compile('<span class="searchTerm">|</span>')
        tmp=str(showList[n][1])
        tmp=s.sub('',str(showList[n][1]))
        addDir(str(n)+'. '+tmp,showList[n][0],6)

def getPageNumber(link):
    p=re.compile('&nbsp; <a href=".*page=(.+?)">Last</a>')
    match=p.findall(link)
    for pageNumber in match:
        return pageNumber

def getFileTypes(url):
    #list filetypes
    p=re.compile('/details/(.*)')
    match=p.findall(url)
    for name in match:
        temp= 'http://www.archive.org/download/'+name+'/'+name+'_files.xml'
    link=getLink(temp)
    tree=BeautifulStoneSoup(link)

    shn=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.shn$)')})
    m3u=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.m3u$)')})
    flac=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.flac$)')})
    mp3=tree.findAll('file', attrs= {"name" : re.compile('(.+?64kb\.mp3$)')})
    vbr=tree.findAll('file', attrs= {"name" : re.compile('(.+?vbr\.mp3$)')})

    if len(m3u)>0:
        addDir('.m3u Playlists',temp,7)
    if len(flac)>0:
        addDir('1. Flac Files',temp,7)
    if len(mp3)>0:
        addDir('2. VBR mp3',temp,7)
    if len(vbr)>0:
        addDir('3. 64kB mp3',temp,7)
    if len(shn)>0:
        addDir('1. Shorten Files',temp,7)

def getTracks(url,rx):
    p=re.compile('(http://www.archive.org/download.*)/.*')
    match=p.findall(url)
    for name in match:
        temp=name
    link=getLink(url)
    tree=BeautifulStoneSoup(link)
    trackList=tree.findAll('file', attrs= {"name" : re.compile(rx)})
    for i in range(len(trackList)):
        url=temp+'/'+trackList[i]['name']
        name=trackList[i].title.contents[0]
        artist=trackList[i].creator.contents[0]
        album=trackList[i].album.contents[0]
        track=trackList[i].track.contents[0]
        addLink(name,url,artist,album,track)
    

######################################################################

site='http://www.archive.org/browse.php?collection=etree&field=%2Fmetadata%2Fcreator'
#site='file:///C:/Program Files/XBMC/plugins/music/archive.org_del.later/artistList.php.html'
params=get_params()
url=None
name=None
mode=None

try:
    url=urllib.unquote_plus(params["url"])
except:
    pass
try:
    name=urllib.unquote_plus(params["name"])
except:
    pass
try:
    mode=int(params["mode"])
except:
    pass

if mode==None or url==None or len(url)<1:
    addDir('Artists A-Z',site,0)
    addDir('Browse all Artists',site,1)

if mode==0:
    addDir('0-9, et al.','0',3)
    bet='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    for i in range(0,26):
        addDir(bet[i],bet[i],3)
if mode==1:
    link=getLink(site)
    bandList=getBandList(link)
    showBandList(bandList)
if mode==2:
    print 'search'
if mode==3:
    link=getLink(site)
    bandList=getBandList(link)
    bandList=alphList(bandList,url)
    showBandList(bandList)
if mode==4:
    link=getLink('http://www.archive.org'+url)
    pageNumber=getPageNumber(link)
    if pageNumber==None:
        getShowList(link)
    else:
        for i in range(1,int(pageNumber)+1):
            addDir('Page '+str(i),url+'&page='+str(i),5)    
if mode==5:
    link=getLink('http://www.archive.org'+url)
    getShowList(link)
if mode==6:
    getFileTypes(url)
if mode==7:
    if name=='.m3u Playlists':
        p=re.compile('(http://www.archive.org/download.*)/.*')
        match=p.findall(url)
        for name in match:
            temp=name
        link=getLink(url)
        tree=BeautifulStoneSoup(link)
        m3u=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.m3u$)')})
        for i in range(len(m3u)):
            addLink(m3u[i].format.contents[0].string,temp+'/'+m3u[i]['name'],"","")
    elif name=='1. Flac Files':
        getTracks(url,'(.+?\.flac$)')
    elif name=='1. Shorten Files':
        getTracks(url,'(.+?\.shn$)')
    elif name=='2. VBR mp3':
        getTracks(url,'(.+?vbr\.mp3$)')
    elif name=='3. 64kB mp3':
        getTracks(url,'(.+?64kb\.mp3$)')
'''if mode==8:
    p=re.compile('(http://www.archive.org/download.*)/.*')
    match=p.findall(url)
    for name in match:
        temp=name
    link=getLink(url)
    tree=BeautifulStoneSoup(link)
    flac=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.flac$)')})
    for i in range(len(flac)):
        url=temp+'/'+flac[i]['name']
        name=flac[i].track.contents[0]+'. '+flac[i].title.contents[0]
        addLink(name,url)'''



xbmcplugin.endOfDirectory(int(sys.argv[1]))
(This post was last modified: 2009-06-09 02:03 by tylerh.)
find quote
w3__ Offline
Senior Member
Posts: 130
Joined: Jul 2008
Reputation: 0
Post: #10
Yo, awesome! Can you guys put this into svn repo?
find quote
tylerh Offline
Junior Member
Posts: 2
Joined: Jun 2009
Reputation: 0
Post: #11
Update to my previous post. I've solved the issue with m3u playlists! Cool

Line 212:
Code:
    addLink(m3u[i].format.contents[0],temp+'/'+m3u[i]['name'],"","")
is missing an empty string parameter. It should be changed to:
Code:
    addLink(m3u[i].format.contents[0],temp+'/'+m3u[i]['name'],"","","")
The final ,"" is all that was missing.

So here is the full, working code. Just replace the default.py file contents in your Live Music Archive plugin folder with what is below.

[EDIT:] I should mention, I also added two calls to the .replace method to correct for html encoded &amp; (&) and &gt; (>) symbols in formatting the track names. I don't expect that will cause any issues, but just in case, you can find the change on line #140 if you notice problems on the flac/mp3 track listing pages.

Code:
#Live Music Archive (archive.org) default.py
#by rwparris2

import urllib,urllib2,re
import xbmcplugin,xbmcgui
from BeautifulSoup import BeautifulStoneSoup, BeautifulSoup

xbmcplugin.setContent(int(sys.argv[1]), 'songs')
xbmcplugin.addSortMethod(int(sys.argv[1]), 7)
xbmcplugin.addSortMethod(int(sys.argv[1]), 9)


def getLink(site):
    req = urllib2.Request(site)
    req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')
    response = urllib2.urlopen(req)
    link=response.read()
    response.close()
    return link

def addDir(name,url,mode):
    u=sys.argv[0]+"?url="+urllib.quote_plus(url)+"&mode="+str(mode)+"&name="+urllib.quote_plus(name)
    ok=True
    liz=xbmcgui.ListItem(name)
    liz.setInfo( type="Music", infoLabels={ "Title": name } )
    ok=xbmcplugin.addDirectoryItem(handle=int(sys.argv[1]),url=u,listitem=liz,isFolder=True)
    return ok

def addLink(name,url,artist,album,track):
    ok=True
    liz=xbmcgui.ListItem(name)
    liz.setInfo( type="Music", infoLabels={"Title": name,"Album":album,"Artist":artist, "Track":track } )
    ok=xbmcplugin.addDirectoryItem(handle=int(sys.argv[1]),url=url,listitem=liz)
    return ok

def get_params():
    param=[]
    paramstring=sys.argv[2]
    if len(paramstring)>=2:
        params=sys.argv[2]
        cleanedparams=params.replace('?','')
        if (params[len(params)-1]=='/'):
            params=params[0:len(params)-2]
        pairsofparams=cleanedparams.split('&')
        param={}
        for i in range(len(pairsofparams)):
            splitparams={}
            splitparams=pairsofparams[i].split('=')
            if (len(splitparams))==2:
                param[splitparams[0]]=splitparams[1]
    return param

######################################################################
def getBandList(link):
    bandList=[]
    p=re.compile('<a href="(.+?)">(.+?)</a>.+?<a href="(.+?)">(.+?)</a>')
    match=p.findall(link)
    for band in match:
        bandList.append(band)
    return bandList

def alphList(bandList,letter):
    alphList=[]
    if letter=='0':
        num='.0123456789'
        for n in range(0,10):
            for i in range(len(bandList)):
                if bandList[i][1][0]==num[n]:
                    alphList.append(bandList[i])
    else:
        for i in range(len(bandList)):
            if bandList[i][1][0]==letter:
                alphList.append(bandList[i])
    return alphList

def showBandList(bandList):
    i=0
    while(i<len(bandList)):
        addDir(bandList[i][1]+'  ('+bandList[i][3]+')',bandList[i][2],4)
        i+=1
    return bandList

def getShowList(link):
    tree=BeautifulSoup(link)
    d=tree.findAll('a', 'titleLink')
    showList=[]
    for i in range(len(d)):
        p=re.compile('href="(.+?)">.*(Live at.+?)</a>')
        match=p.findall(str(d[i]))
        for show in match:
            showList.append(show)
    for n in range(len(showList)):
        s=re.compile('<span class="searchTerm">|</span>')
        tmp=str(showList[n][1])
        tmp=s.sub('',str(showList[n][1]))
        addDir(str(n)+'. '+tmp,showList[n][0],6)

def getPageNumber(link):
    p=re.compile('&nbsp; <a href=".*page=(.+?)">Last</a>')
    match=p.findall(link)
    for pageNumber in match:
        return pageNumber

def getFileTypes(url):
    #list filetypes
    p=re.compile('/details/(.*)')
    match=p.findall(url)
    for name in match:
        temp= 'http://www.archive.org/download/'+name+'/'+name+'_files.xml'
    link=getLink(temp)
    tree=BeautifulStoneSoup(link)

    shn=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.shn$)')})
    m3u=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.m3u$)')})
    flac=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.flac$)')})
    mp3=tree.findAll('file', attrs= {"name" : re.compile('(.+?64kb\.mp3$)')})
    vbr=tree.findAll('file', attrs= {"name" : re.compile('(.+?vbr\.mp3$)')})

    if len(m3u)>0:
        addDir('.m3u Playlists',temp,7)
    if len(flac)>0:
        addDir('1. Flac Files',temp,7)
    if len(mp3)>0:
        addDir('2. VBR mp3',temp,7)
    if len(vbr)>0:
        addDir('3. 64kB mp3',temp,7)
    if len(shn)>0:
        addDir('1. Shorten Files',temp,7)

def getTracks(url,rx):
    p=re.compile('(http://www.archive.org/download.*)/.*')
    match=p.findall(url)
    for name in match:
        temp=name
    link=getLink(url)
    tree=BeautifulStoneSoup(link)
    trackList=tree.findAll('file', attrs= {"name" : re.compile(rx)})
    for i in range(len(trackList)):
        url=temp+'/'+trackList[i]['name']
        name=trackList[i].title.contents[0].replace("&amp;", "&").replace("&gt;", ">")
        artist=trackList[i].creator.contents[0]
        album=trackList[i].album.contents[0]
        track=trackList[i].track.contents[0]
        addLink(name,url,artist,album,track)
    

######################################################################

site='http://www.archive.org/browse.php?collection=etree&field=%2Fmetadata%2Fcreator'
#site='file:///C:/Program Files/XBMC/plugins/music/archive.org_del.later/artistList.php.html'
params=get_params()
url=None
name=None
mode=None

try:
    url=urllib.unquote_plus(params["url"])
except:
    pass
try:
    name=urllib.unquote_plus(params["name"])
except:
    pass
try:
    mode=int(params["mode"])
except:
    pass

if mode==None or url==None or len(url)<1:
    addDir('Artists A-Z',site,0)
    addDir('Browse all Artists',site,1)

if mode==0:
    addDir('0-9, et al.','0',3)
    bet='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    for i in range(0,26):
        addDir(bet[i],bet[i],3)
if mode==1:
    link=getLink(site)
    bandList=getBandList(link)
    showBandList(bandList)
if mode==2:
    print 'search'
if mode==3:
    link=getLink(site)
    bandList=getBandList(link)
    bandList=alphList(bandList,url)
    showBandList(bandList)
if mode==4:
    link=getLink('http://www.archive.org'+url)
    pageNumber=getPageNumber(link)
    if pageNumber==None:
        getShowList(link)
    else:
        for i in range(1,int(pageNumber)+1):
            addDir('Page '+str(i),url+'&page='+str(i),5)    
if mode==5:
    link=getLink('http://www.archive.org'+url)
    getShowList(link)
if mode==6:
    getFileTypes(url)
if mode==7:
    if name=='.m3u Playlists':
        p=re.compile('(http://www.archive.org/download.*)/.*')
        match=p.findall(url)
        for name in match:
            temp=name
        link=getLink(url)
        tree=BeautifulStoneSoup(link)
        m3u=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.m3u$)')})
        for i in range(len(m3u)):
            addLink(m3u[i].format.contents[0],temp+'/'+m3u[i]['name'],"","","")
    elif name=='1. Flac Files':
        getTracks(url,'(.+?\.flac$)')
    elif name=='1. Shorten Files':
        getTracks(url,'(.+?\.shn$)')
    elif name=='2. VBR mp3':
        getTracks(url,'(.+?vbr\.mp3$)')
    elif name=='3. 64kB mp3':
        getTracks(url,'(.+?64kb\.mp3$)')
'''if mode==8:
    p=re.compile('(http://www.archive.org/download.*)/.*')
    match=p.findall(url)
    for name in match:
        temp=name
    link=getLink(url)
    tree=BeautifulStoneSoup(link)
    flac=tree.findAll('file', attrs= {"name" : re.compile('(.+?\.flac$)')})
    for i in range(len(flac)):
        url=temp+'/'+flac[i]['name']
        name=flac[i].track.contents[0]+'. '+flac[i].title.contents[0]
        addLink(name,url)'''



xbmcplugin.endOfDirectory(int(sys.argv[1]))
(This post was last modified: 2009-06-10 23:08 by tylerh.)
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #12
Thanks for your work guys.

It needs a complete rewrite, plus archive.org has tons of xbmc-friendly content that should go in here.

I'll get around to it one of these days.

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
find quote
ChinaRider Offline
Junior Member
Posts: 15
Joined: Oct 2008
Reputation: 0
Post: #13
Thanks rwparris2 for the plugin and thanks tylerh for the modifications that made it work at this point in time.

I love the Live Music Archive and am ecstatic to have this functionality on my xbmc!!!

You guys are the best!
find quote
w3__ Offline
Senior Member
Posts: 130
Joined: Jul 2008
Reputation: 0
Post: #14
This one is also awesome, looking forward to more revisions! Thanks guys!
find quote