IMDB not accepting certain useragent strings? - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32) +--- Forum: Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=60) +--- Thread: IMDB not accepting certain useragent strings? (/showthread.php?tid=65882) |
IMDB not accepting certain useragent strings? - yee379 - 2010-01-03 just noticed today that i couldn't scrap any movies from my ubuntu system: looking at the logs i get: Code: 16:51:57 T:140189498325328 M:835964928 DEBUG: FileCurl::Open(0x7fffd6390d08) http://akas.imdb.com/find?s=tt;q=the%20warlords%20(2007) attempting a wget: Code: $ wget 'http://akas.imdb.com/find?s=tt;q=the%20warlords%20(2007)' --2010-01-02 16:52:32-- http://akas.imdb.com/find?s=tt;q=the%20warlords%20(2007) but doing a wget with -U: Code: $ wget -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.14) Gecko/20080418 Ubuntu/7.10 (gutsy) Firefox/2.0.0.14' 'http://akas.imdb.com/find?s=tt;q=the%20warlords%20(2007)' also wget works fine on my mac... is there an option somewhere where we can overload the useragent strings that xbmc/imdb scrapper uses? - jgora - 2010-01-03 I am really new to xbmc and all - but I think I may have a similar problem to yourself. I'm finding that when I add a source for movies to xbmc and use imdb as a scrapper it doesn't seem to add anything to the library after scanning for about 60 seconds. However when I change the scrapper to tvdb.com it seems to be able to dload all the information. Is there a limit as to how much you can use/download info from IMDB?! Is there a fix for this? Thanks - delirial - 2010-01-03 That is correct. Getting the following for every movie that it tries to scrap since around 9:00PM yesterday: 00:15:06 T:2619337584 M:2954203136 ERROR: CFileCurl::CReadState::Open, didn't get any data from stream. Changing the scrapper to themoviedb.org seems to work (though I like IMDB better). regards, del - nick8539 - 2010-01-03 I'm Having the same problem with IMDB on AppleTV. Is imdb scraping down? I can scrape with tmdb but its not as big ad imdb. - plankton88 - 2010-01-03 Having trouble too. Tried to search for help in others area, but a nothing right now. Maybe I should update..? Using A build from November I think. - Nuka1195 - 2010-01-03 try adding |User-Agent={your valid user agent} to end of the urls in imdb.xml urlencoded of course - rebaker501 - 2010-01-03 stupid question.....what is a valid user agent? - delirial - 2010-01-03 rebaker501, 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.14) Gecko/20080418 Ubuntu/7.10 (gutsy) Firefox/2.0.0.14' is an example of a valid one. Basically, the user-agent is how your browser identifies itself when requesting a website. IE: When you go to download firefox, you get the link for your platform (Mac, Win, Linux) automatically. That's because the website uses the user-agent to determine what is the right download. If anyone gets this to work, please let us know. del Seeing same from Vista x64 - aquariumdrinker - 2010-01-03 No luck after adding my Firefox user agent string to the end of the url. My situation is somewhat similar to that described above (some time this afternoon, IMDB stopped yielding any results). Code: 03:31:59 T:4316 M:4294967295 DEBUG: CVideoDatabase::GetMovieId (F:\Movies\12 Angry Men.m4v), query = select idMovie from movie where idFile=1 - yee379 - 2010-01-03 Nuka1195 Wrote:try adding |User-Agent={your valid user agent} to end of the urls in imdb.xml could you clarify what you mean please? i tried putting in: Code: <RegExp input="$$1" output="<url>http://akas.imdb.com/find?s=tt;q=\1$$4</url>%7CUser-Agent=Mozilla%2F5.0%20(X11%3B%20U%3B%20Linux%20i686%3B%20en-US%3B%20rv%3A1.8.1.14)%20Gecko%2F20080418%20Ubuntu%2F7.10%20(gutsy)%20Firefox%2F2.0.0.14" dest="3"> but it still doesn't work better still, could someone update the imdb.xml file on svn? cheers, - spiff - 2010-01-03 Code: <RegExp input="$$1" output="<url>http://akas.imdb.com/find?s=tt;q=\1$$4|User-Agent=Mozilla%2F5.0%20(X11%3B%20U%3B%20Linux%20i686%3B%20en-US%3B%20rv%3A1.8.1.14)%20Gecko%2F20080418%20Ubuntu%2F7.10%20(gutsy)%20Firefox%2F2.0.0.14</url>" dest="3"> - delirial - 2010-01-03 thanks spiff! Seems to be working now. - CloudDweller - 2010-01-03 I'm totally new to all this editing XML stuff so can someone please either upload their edited XML or give me noobs guide as to what exactly I need to do as I can't get IMDB scraping to work? Thanks - rufus210 - 2010-01-03 Spiff: Thanks, this works. ChrisWad: find system/scrapers/video/imdb.xml where you installed XBMC to. Open up the file with a text editor. Line 46 should contain "http://akas.imdb.com/find" (it's the only instance of "find" in the file). Replace the original line with the one Spiff posted. - jgora - 2010-01-03 does that mean you will need to have mozilla firefox installed? - apologies if that is an obvious question |