New music video scraper with fanarts support

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
sam_1421 Offline
Junior Member
Posts: 34
Joined: Apr 2012
Reputation: 1
Post: #1
Hi,

I'm currently working on a music video scraper retrieving data from Last.fm and MusicBrainz and fanarts from HTBackdrops. In fact, it's almost complete. I only have a small problem : I can't get access the "fanarts" tab, despite the fact that in the debugger, fanarts are clearly found and correctly parsed,

Here is the XML (I hid the real url, beacause it's only a test server and can't handle a lot of requests):
Code:
<?xml version="1.0" encoding="utf-8"?><scraper framework="1" date="2012-01-18" name="Last.fm Music Video Scraper" content="musicvideos" thumb="icon.png" language="en">
    <CreateSearchUrl dest="3">
        <RegExp input="$$1" output="&lt;url&gt;http://ws.audioscrobbler.com/2.0/?method=track.search&amp;track=\1&amp;api_key=b25b959554ed76058ac220b7b2e0a026&lt;/url&gt;" dest="3">
            <expression noclean="1" />
        </RegExp>
    </CreateSearchUrl>
    <GetSearchResults dest="3">
        <RegExp input="$$5" output="&lt;results&gt;\1&lt;/results&gt;" dest="3">
            <RegExp input="$$8" output="&lt;entity&gt;\1&lt;/entity&gt;" dest="5">
                <RegExp input="$$1" output="&lt;title&gt;\2 - \1&lt;/title&gt;" dest="8">
                    <expression>&lt;track&gt;\s*&lt;name&gt;([^&lt;]*)&lt;/name&gt;\s+&lt;artist&gt;([^&lt;]*)&lt;/artist&gt;</expression>
                </RegExp>
                <RegExp input="$$1" output="&lt;url&gt;http://xxxxx.com/scraper/search.php?artist=\2&amp;track=\1&lt;/url&gt;" dest="8+">
                    <expression encode="1,2">&lt;track&gt;\s*&lt;name&gt;([^&lt;]*)&lt;/name&gt;\s+&lt;artist&gt;([^&lt;]*)&lt;/artist&gt;</expression>
                </RegExp>
                <expression repeat="yes" noclean="1" />
            </RegExp>
            <expression noclean="1" />
        </RegExp>
    </GetSearchResults>
    <GetDetails dest="3">
        <RegExp input="$$5" output="&lt;details&gt;\1&lt;/details&gt;" dest="3">
            <RegExp input="$$1" output="\1" dest="7">
                <expression noclean="1">&lt;artist&gt;(.*)&lt;/artist&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;title&gt;\1&lt;/title&gt;" dest="5">
                <expression>&lt;track&gt;(.*)&lt;/track&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;artist&gt;\1&lt;/artist&gt;" dest="5+">
                <expression>&lt;artist&gt;(.*)&lt;/artist&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;year&gt;\1&lt;/year&gt;" dest="5+">
                <expression>&lt;year&gt;(.*)&lt;/year&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;album&gt;\1&lt;/album&gt;" dest="5+">
                <expression>&lt;album&gt;(.*)&lt;/album&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;thumb&gt;\1&lt;/thumb&gt;" dest="5+">
                <expression>&lt;thumb&gt;(.*)&lt;/thumb&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;genre&gt;\1&lt;/genre&gt;" dest="5+">
                <expression repeat="yes">&lt;genre&gt;(.*)&lt;/genre&gt;</expression>
            </RegExp>
            <RegExp input="$$1" output="&lt;plot&gt;\1&lt;/plot&gt;" dest="5+">
                <expression>&lt;plot&gt;(.*)&lt;/plot&gt;</expression>
            </RegExp>
            <RegExp input="$$7" output="&lt;chain function=&quot;GetHTBFanart&quot;&gt;\1&lt;/chain&gt;" dest="5+">
                <expression/>
            </RegExp>
            <expression noclean="1" />
        </RegExp>
        
    </GetDetails>
    <GetHTBFanart dest="5">
        <RegExp input="$$1" output="&lt;details&gt;&lt;url function=&quot;ParseHTBFanart&quot; post=&quot;yes&quot; cache=&quot;htb-images-\1.xml&quot;&gt;http://htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/searchXML?keywords=\1&amp;default_operator=and&amp;aid=1,5&lt;/url&gt;&lt;/details&gt;" dest="5">
            <expression noclean="1" />
        </RegExp>
    </GetHTBFanart>
    <ParseHTBFanart dest="5">
        <RegExp input="$$13" output="&lt;details&gt;&lt;fanart&gt;\1&lt;/fanart&gt;&lt;/details&gt;" dest="5">
            <RegExp input="$$1" output="&lt;thumb preview=&quot;http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/\1/thumbnail&quot;&gt;http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/\1/fullsize&lt;/thumb&gt;" dest="13">
                <expression repeat="yes" noclean="1">&lt;id&gt;([^&lt;]+)&lt;/id&gt;\n[^&lt;]+&lt;aid&gt;1&lt;/aid&gt;</expression>
            </RegExp>
            <expression noclean="1">(.+)</expression>
        </RegExp>
    </ParseHTBFanart>
        
</scraper>

Here is the what comes out from the debugger for the video "The Veronicas - Untouched" :
Code:
02:31:13 T:8304   DEBUG: scraper: GetDetails returned <details><artist>The Veronicas</artist><album>Untouched</album><thumb>http://userserve-ak.last.fm/serve/300x300/58306283.png</thumb><genre>Pop / Female Vocalists / Dance / The Veronicas / Australian</genre><plot>&quot;Untouched&quot; is the second single by The Veronicas from their sophomore album, Hook Me Up. It was released in December 2007 to Australia. It is also the first single from the same album in North America and Europe. The song is written by Jess and Lisa and also has writing credits from Toby Gad. According to the Untouched Songfacts, the song is &quot;about a long distance relationship and having to interact over the technology of today .&quot;
                                             It peaked at #2 on the Australian Top 50. After being officially released to US radio in April 2008, it took six months before the single took off. By Christmas, the single had been added to Z100, the biggest station in the US. In Canada, it took the single a little longer to take off, but eventually outpeaked the US' peak. The song peaked at #17 in the US, and #5 in Canada. Due to its success, the single is planned to be released in Europe. It's already charted in Finland at #20, and the Czech Republic at #12. The single had also peaked at #9 in New Zealand and #71 in Chile. It is said to be their biggest single world wide and their breakthrough into the US market. The single also became the 12th most played song on American pop radio for the week of February 15, 2009.
                                                    
                                            User-contributed text is available under the Creative Commons By-SA License and may also be available under the GNU FDL.</plot><chain function="GetHTBFanart">The Veronicas</chain></details>
02:31:13 T:8304   DEBUG: scraper: GetHTBFanart returned <details><url function="ParseHTBFanart" post="yes" cache="htb-images-The Veronicas.xml">http://htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/searchXML?keywords=The Veronicas&default_operator=and&aid=1,5</url></details>
02:31:13 T:8304   DEBUG: FileCurl::Open(0025A8D8) http://htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/searchXML
02:31:13 T:8304   DEBUG: scraper: ParseHTBFanart returned <details><fanart><thumb preview="http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/1682/thumbnail">http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/1682/fullsize</thumb><thumb preview="http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/5525/thumbnail">http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/5525/fullsize</thumb><thumb preview="http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/5526/thumbnail">http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/5526/fullsize</thumb><thumb preview="http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/5527/thumbnail">http://www.htbackdrops.com/api/7681a907c805e0670330c694e788e8e8/download/5527/fullsize</thumb></fanart></details>
02:31:13 T:8304   DEBUG: Thread CVideoInfoDownloader 8304 terminating
02:31:13 T:8440   DEBUG: VideoInfoScanner: Adding new item to musicvideos:L:\Music Videos\Old - new\The Veronicas - Untouched.avi

So yeah, clearly the scraper finds some fanarts, but I can't access them, the "fanart" tab is unclickable...

Do you know where my error is, or what the problem is?

Thanks!

Sam
(This post was last modified: 2012-04-29 17:48 by sam_1421.)
find quote
sam_1421 Offline
Junior Member
Posts: 34
Joined: Apr 2012
Reputation: 1
Post: #2
In fact I just noticed the fanart gets loaded, and it appears in the background when hovering a music video, just like when hovering movies! Whoo!

So the only and last detail to make this a final version is : how can I allow the user to select his fanart? The fanart tab is unclickable and it always load the first fanart in the list, even if there are more than one...
(This post was last modified: 2012-04-29 17:51 by sam_1421.)
find quote
RandomNinjaAtk Offline
Member
Posts: 77
Joined: Jun 2011
Reputation: 1
Location: USA -> Maryland
Post: #3
I wish I could help you out on this. But regardless of that, I'm excited for a Music Video Scraper that works! Keep up the good work, if you need someone for testing. I wouldn't mind testing it for you. I have a collection of music videos, roughly 40, but I'm looking to grow it.

"Live to Win!"
www.stevenohare.com - Personal Blog
www.oharesites.com - Domain Names, Web Hosting, Email Hosting and more
find quote
sam_1421 Offline
Junior Member
Posts: 34
Joined: Apr 2012
Reputation: 1
Post: #4
I changed it a little, but still have the fanart problem ...

I will PM you RandomNinjaAtk to send you the addon so you can test it! I'm sorry for everyone else, I can't make this public yet, it's based on a server which can't really take much demand :/

If the feedback is good I'll try to make it available for everyone though Wink
find quote
RandomNinjaAtk Offline
Member
Posts: 77
Joined: Jun 2011
Reputation: 1
Location: USA -> Maryland
Post: #5
Sounds good.

"Live to Win!"
www.stevenohare.com - Personal Blog
www.oharesites.com - Domain Names, Web Hosting, Email Hosting and more
find quote
G elly N Offline
Member
Posts: 77
Joined: Dec 2011
Reputation: 0
Post: #6
the fanart come from the music liabry as dose the artist pic, so you will only get the fanart if you have that artist in your music liabary
find quote
sam_1421 Offline
Junior Member
Posts: 34
Joined: Apr 2012
Reputation: 1
Post: #7
I confirm it doesn't require having the artist inthe music library, as I don't have any music added XBMC and it still scrapes the fanarts
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #8
(stupid)
(This post was last modified: 2012-04-30 18:23 by spiff.)
find quote
RockerC Offline
Senior Member
Posts: 246
Joined: May 2011
Reputation: 3
Post: #9
(2012-04-29 08:48)sam_1421 Wrote:  I'm currently working on a music video scraper retrieving data from Last.fm and MusicBrainz and fanarts from HTBackdrops.
Good stuff! Will you please consider adding support for Music FanArt from the fanart.tv website as well? Big Grin

http://fanart.tv
http://fanart.tv/music-fanart/
http://fanart.tv/api-docs/music-api/

Maybe make fanart.tv primary for FanArt backgrounds/backdrops and only use HTBackdrops.com as backup?

fanart.tv have more active contributes and higher standards, and it features not only FanArt for music!

They also have ClearLOGO and Album Cover which could maybe be used for music videos too Wink
(This post was last modified: 2012-05-02 13:17 by RockerC.)
find quote
st graveyard Offline
Fan
Posts: 560
Joined: Apr 2011
Reputation: 3
Post: #10
Wow, this sounds indeed exciting.

Sam, if you want other testers let me know. I too have some music videos that are not yet implemented in XBMC (because of bad scrapers). I hope you get this thing solved (I'm not a coder, sadly Sad ).

Keep us informed.

Good luck,
Grave
find quote
RandomNinjaAtk Offline
Member
Posts: 77
Joined: Jun 2011
Reputation: 1
Location: USA -> Maryland
Post: #11
I don't know if this is being worked on anymore.... sam_1421 did provide me with a version to test, so I did and provided feedback via PM. But haven't heard a word since then and that was on April 30th.. I hope everything is okay and it would be great to see the scraper completed.

--Ninja

"Live to Win!"
www.stevenohare.com - Personal Blog
www.oharesites.com - Domain Names, Web Hosting, Email Hosting and more
find quote
CaBrA Offline
Junior Member
Posts: 16
Joined: Sep 2011
Reputation: 0
Post: #12
Is there any other scraper than Yahoo Music for Music Vids?
find quote
sialivi Offline
Posting Freak
Posts: 1,079
Joined: Dec 2008
Reputation: 9
Location: Sweden
Post: #13
(2012-06-14 15:13)CaBrA Wrote:  Is there any other scraper than Yahoo Music for Music Vids?

RizThon's Last.fm Music Video Scraper, it's not in the repositories though so you have to download it from the forum thread. http://forum.xbmc.org/showthread.php?tid...#pid995028
(This post was last modified: 2012-06-14 16:41 by sialivi.)
find quote
CaBrA Offline
Junior Member
Posts: 16
Joined: Sep 2011
Reputation: 0
Post: #14
thank you
find quote
olympia Offline
Team-XBMC Member
Posts: 2,492
Joined: May 2008
Reputation: 32
Post: #15
(2012-04-30 03:20)G elly N Wrote:  the fanart come from the music liabry as dose the artist pic, so you will only get the fanart if you have that artist in your music liabary

(2012-06-14 16:39)sialivi Wrote:  
(2012-06-14 15:13)CaBrA Wrote:  Is there any other scraper than Yahoo Music for Music Vids?

RizThon's Last.fm Music Video Scraper, it's not in the repositories though so you have to download it from the forum thread. http://forum.xbmc.org/showthread.php?tid...#pid995028

look around in the repository then...
It's not the one from RizThon, but there is something called last.fm...
find quote