Creating Music Video scraper

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Question  Creating Music Video scraper
Post: #1
I'd like to create a Music Video scraper from last.fm website.

I've read http://wiki.xbmc.org/index.php?title=Wri...pers_guide but I don't know:
  • if it's up-to-date for XBMCv10 (am I supposed to create an addon or just a simple xml file as described?)
  • what are the things specifics to Music Video scrapers (looks like I should use content="mvid", but I don't know what tags to use for the GetDetails parts)
  • how to test it: if I copy my scraper file to system\scrapers\video, it can't see it anywhere when I'm running XBMC (I also already have several scrapers in this folder that ships with XBMC, and I also don't see them in XBMC)
find quote
mkortstiege Offline
Team-XBMC Developer
Posts: 2,921
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #2
You'll have to addon-ify it. Please check http://wiki.xbmc.org/index.php?title=Add-on_development.

For an example addon layout (musicvideo scraper) please have a look at https://github.com/xbmc/xbmc/tree/master...omusic.com

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Thumbs Up   
Post: #3
Thanks a lot vdrfan for the info. It's quite simple to addon-ify it.

Do you know if it's possible to link several scrapers together? eg if Yahoo Music doesnt' return anything, try last.fm? But it probably shouldn't be the job of the addon to do that anyway, it's more XBMC that should be able to try several scrapers if one doesn't find anything (or enough info).
find quote
mkortstiege Offline
Team-XBMC Developer
Posts: 2,921
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #4
Right, currently there's no such functionality besides a very basic language based fallback when it comes to url nfo scraping. You would have to do it all within one addon.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Post: #5
While you're at it, maybe you'll be able to help me with my scraper...

When I use the C# ScraperXML Editor, it looks like everything is fine and it returns
<details><title>I Hate You</title><artist>2NE1</artist></details>

but when I use it in XBMC, it says it can't parse the result

DEBUG: FileCurl::Open(039EF168) http://ws.audioscrobbler.com/2.0/?method...E1&track=I Hate You
WARNING: XFILE::CFileCurl::CReadState::FillBuffer: curl failed with code 52
ERROR: CFileCurl::CReadState::Open, didn't get any data from stream.
DEBUG: FileCurl::Close(039EF168) http://ws.audioscrobbler.com/2.0/?method...E1&track=I Hate You
ERROR: ADDON::CScraper::Run: Unable to parse web site

Here's the xml file http://pastebin.com/1nsWYzzr

Thanks.
find quote
mkortstiege Offline
Team-XBMC Developer
Posts: 2,921
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #6
Looks like you need to encode the artist and track string passed to the url.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Thumbs Up   
Post: #7
vdrfan Wrote:Looks like you need to encode the artist and track string passed to the url.
Wow, thanks a lot for the tip! You saved me from getting insane! At 1st I didn't get it, but then I realized the spaces should be %20 in the address...I was just missing a simple encode="1,2".

Writing scrapers is really simple (at least if you want to do simple things), but it's just sooo easy to make tiny stupid mistakes that eventuallly it's not that simple and takes some trial and error.
find quote
mkortstiege Offline
Team-XBMC Developer
Posts: 2,921
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #8
Glad to hear you were able to solve it Wink

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Post: #9
The last.fm scraper works fine (at least with XBMC 10.1, don't know if it'll work with the new version) and I feel it finds more info than the one from Yahoo (eg for Asian artists), so I can submit it for others to use (here is the new xml file http://pastebin.com/LN86uRuH). Before doing so I'd like to polish it, so I still have a few questions:
  • what are all the possible tags for the GetDetails part? Is the Yahoo! Music Video plugin already using them all? eg it uses no "plot" tag or anything to store info about the song
  • the scraper works fine, the library contains the correct info, the only thing is that when XBMC is updating the DB, the name of the artist and song displayed in the progress bar at the top still contains characters encoded for the web, eg %20, so it's not easily readable (again, in file and library mode everything is displayed correctly). I feel it's again an encoding issue...
  • something not necessarily related to the scraper: it looks like the artist and album info are not retrieved for music videos (eg if I have a music from an artist and a music video of that same artist, I'll have a nice background image - I guess from htbackdrops - but if I only have a music video from that artist and no music, then I have the default background image from the theme). Am I doing something wrong, or XBMC doesn't run the artist and album addons for music videos?
  • definitely not related: in XBMC I see Music, Movies, Videos, but not directly Music Videos, I have to go under Videos to see it. Is there a setting for it somewhere?
Thanks.
find quote
mkortstiege Offline
Team-XBMC Developer
Posts: 2,921
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #10
1. a music-video can basically handle all the video info tags movies do + track, track and artist. (depends on the skin which information are shown)
2. caused by the encode you added as the strings are also used as the title.
3. should work as a normal fanart that's fetched based on the artist string. The artist fanart from music library are not used at this point (afaik).
4. depends on the skin. Confluence should show a music-video sub menu on the videos home item.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
vdrmidas Offline
Junior Member
Posts: 3
Joined: Jan 2012
Reputation: 0
Post: #11
Thank you for developing the scraper. Yahoo finds 27 out of 83 music vids whereas your lastFM scraper finds 82! Finally i can play the vids now in party mode though i do not understand at all why party mode is limited to db entries... Unfortunately on the other hand the scraper doesnt correctly fill the db. There is absolutely no information Smile No titles no artists no additional info. I have tested under Debian with a recent pre eden git snapshot as well as under xp with the stable release. This might be due to the fact that pastebin is not the best to paste code. I had to manually remove newlines which i hopefully did correctly. But it may be a source of evil Wink Could you maybe attach a zipped addon dir here?

I will subscribe to this thread. Do not hesitate, i'll have an exam at the end of next week Wink

greets

midas
find quote
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Post: #12
Glad to know it's useful to someone else than just me!

I uploaded the zipped addon-ified scraper http://www.mediafire.com/?n3yc8g1u20qjjqi and tried it in XBMC 10.1 and 11 beta 1 on Win7. Please test if it works fine on XP and Debian. If there's no problem I'll ask the scraper to be added to the XBMC repo.

FYI:
I added the plot tag (#1 from previous post
I fixed the encoding issue (#2 from previous post)
find quote
RizThon Offline
Junior Member
Posts: 11
Joined: Jan 2012
Reputation: 0
Post: #13
Concerning #3 from previous post, it indeed looks like XBMC doesn't run the artist and album addons for music videos. But if that info is available (ie retrieved previously through the music), it will be available for the music videos (at least backdrop and artist image).

I don't know how the code is done, but shouldn't it be quite simple to run the artist and album addons for the music videos also? Should I add it to the bug tracker (looks like no one asked for that yet)?
find quote
sialivi Offline
Posting Freak
Posts: 1,156
Joined: Dec 2008
Reputation: 9
Location: Sweden
Post: #14
Great work. Works much better than the old scrapers.
find quote
Tarkules Offline
Junior Member
Posts: 17
Joined: Nov 2011
Reputation: 0
Post: #15
RizThon Wrote:Glad to know it's useful to someone else than just me!

I uploaded the zipped addon-ified scraper http://www.mediafire.com/?n3yc8g1u20qjjqi and tried it in XBMC 10.1 and 11 beta 1 on Win7. Please test if it works fine on XP and Debian. If there's no problem I'll ask the scraper to be added to the XBMC repo.

FYI:
I added the plot tag (#1 from previous post
I fixed the encoding issue (#2 from previous post)


Great work mate!
Thank you very much for the efforts, you solved a very annoying problem for me Smile

CHEERS !
find quote
Post Reply