Does XBMC and XBMC Media Center end-users have permission to scrape imdb.com website?

  Thread Rating:
  • 1 Votes - 1 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thread Closed
ultrabrutal Offline
Posting Freak
Posts: 952
Joined: Feb 2005
Reputation: 0
Location: South of Heaven
Question  Does XBMC and XBMC Media Center end-users have permission to scrape imdb.com website?
Post: #1
Just to take one, imdb:

IMDB Conditions

"Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express written consent as noted below."

Do XBMC have this permission? Just curious...
find
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #2
no, i was not aware of those terms (or atleast i could claim so).

not so much any longer. r17032 disables the scraper
find
ultrabrutal Offline
Posting Freak
Posts: 952
Joined: Feb 2005
Reputation: 0
Location: South of Heaven
Post: #3
Shit! What a party pooper I turned out be Sad

I'm afraid that most sites have conditions like this so most scrapers are in fact violating terms of the site
find
theuni Offline
Team-XBMC Communication Manager
Posts: 1,105
Joined: Oct 2007
Reputation: 2
Location: Atlanta, Ga, USA
Post: #4
Quote:We do allow the limited use of robots and crawlers, such as those from certain search engines, with our express written consent. If you are interested in receiving our express written permission to use robots or crawlers on our site, please contact our Licensing Department.

I'd be happy to kick-off communications. I'd think this is the type of project that they would be open to.

Spiff: Let me know if this is something I should look into, or if it's up to the XBMC team to handle.

TheUni
find
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #5
i think we will have to handle this ourself.

thanks for the offer though
find
ultrabrutal Offline
Posting Freak
Posts: 952
Joined: Feb 2005
Reputation: 0
Location: South of Heaven
Post: #6
I'm afraid we eventually have to create our own movie database ala the mymovies database which we can all contribute to Sad
find
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #7
themoviedb.org is just that. i have already written the scraper a few months ago, but as the site had so little info i didn't bother with polishing it.
find
pike Offline
Project Manager
Posts: 4,835
Joined: Sep 2003
Reputation: 29
Location: Sweden
Post: #8
Let me quote Conan O Brien "Nicely Done" Cool

ultrabrutal Wrote:Just to take one, imdb:

IMDB Conditions

"Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express written consent as noted below."

Do XBMC have this permission? Just curious...

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find
ultrabrutal Offline
Posting Freak
Posts: 952
Joined: Feb 2005
Reputation: 0
Location: South of Heaven
Post: #9
Pike, when commitmonitor showed me the commits spiff had done I felt really bad. I use the scrapers also hehe. Of course XBMC cannot violate any licenses, so I totally get spiff.

spiff, I didn't know about themoviedb.org. Do you use an api or really scrape the html? I don't see an imdb link, so you use the movie name as a param?
find
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #10
it's an xml based api.

see r17035
find
mkortstiege Offline
Team-XBMC Developer
Posts: 2,918
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #11
ultrabrutal Wrote:Pike, when commitmonitor showed me the commits spiff had done I felt really bad. I use the scrapers also hehe. Of course XBMC cannot violate any licenses, so I totally get spiff.

spiff, I didn't know about themoviedb.org. Do you use an api or really scrape the html? I don't see an imdb link, so you use the movie name as a param?

A basic themoviedb.org scraper just hit the SVN in revision 17035. Searches and IMDB-ID translation is done via their API.

FYI, it's the new default movie scraper ..

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find
ultrabrutal Offline
Posting Freak
Posts: 952
Joined: Feb 2005
Reputation: 0
Location: South of Heaven
Post: #12
The first thing that springs to mind when looking at the site, is... Who owns the copyright of fanart and posters? I think there is a grey area there also I'm afraid Sad

offtopic: spiff, have looked into the api info. looks easy to use. any indications on speed vs the others?
find
LaTropa64 Offline
Fan
Posts: 649
Joined: Nov 2005
Reputation: 0
Post: #13
No more imdb scraping is going to kill one of the best things about XBMC. Sad

I've added a few things to themoviedb.org when movies were missing fanart but manually entering all the actors and plot details doesn't sound like much fun. I've exported my library as nfo files though so I wonder if there's any chance we could just upload all our nfo's to a central location (like themoviedb.org) for scraping?
find
ultrabrutal Offline
Posting Freak
Posts: 952
Joined: Feb 2005
Reputation: 0
Location: South of Heaven
Post: #14
The nfo's are based on copyright data so they are a violation also I think.

I don't think that manually entering the data is that hard when enough people contribute. It's our best bet.
You can use mymovies.dk but it only has movies which can be bought on media
find
marlboroman1 Offline
Fan
Posts: 316
Joined: Aug 2007
Reputation: 0
Post: #15
I'm would realy like some one to quietly delete this thread so we can carry on in blissfull ignorance as if non of this unpleasantness occured
find
Thread Closed