Using Bing for better scraper searching

So I got tired of the existing scrapers returning incorrect results for about 1/3rd of my movies. It turns out that while the sites we scrape have lots of great data, their search engines range from inaccurate (imdb's) to completely broken (tmdb's). They both choke when your naming doesn't exactly match the official name of the movie and really don't like foreign movies.

Any real search engine can handle these cases just fine. Looking around I found that Bing has a very nice, easy to use developer API for accessing their search results. Google and Yahoo both also have APIs, but they are only for use as part of an AJAX website (Google's FAQ says they'll block you if you scrape their results). The Bing ToU allows "end-user-facing website or application".

Anyway, I edited the existing IMDB scraper to do a Bing search of "site:imdb.com movie (year)" and parse the returned XML. The actual data is still scraped from IMDB, I just changed the search part. For my collection of ~200 movies (hollywood, anime, foreign, etc) Bing got the correct imdb link 100% of the time.

This method could be used for any scraper by replacing "site:imdb.com" with "site:themoviedb.org" or something else.

Question for an XBMC admin: Bing's API requires a AppID, just like TMDB's. I signed up for one personally but would rather not release the scraper using my AppID. Would it be possible for someone @xbmc.org to sign up for an official AppID that can be used? It's an online process that takes about 5 minutes.

Once the AppID is squared away I'll release my bing_imdb.xml file, and post a little guide for how to add bing to other scrapers.

I made this appid: 16E50AB9947899C41433EB944C60174737855036

you dont need api to scrape you know. if im not mistaken the imdb scraper did use google before. i've tested google and its not perfect. there are pages missing and it can have problems with aka titles. whether google is better than imdb all depends on which movies you have i guess. maybe you should try this before completely rule out imdb.

and btw google will only block you for small amount of time if you do like 500 searches in <1min.

Thanks for the AppID Pike. Here are the 2 files for my modification of the imdb scraper. Drop them into the system/scrapers/video directory of your install.

http://www.hackish.org/~rufus/bing_imdb.xml
http://www.hackish.org/~rufus/bing_imdb.png

Can people try it out and report how it works? It works perfectly for me.

To use this for other scrapers simply replace the CreateSearchUrl and GetSearchResults parts with the one from my file. In the CreateSearchUrl replace site:imdb.com with site:somethingelse.com. In the GetSearchResults replace "http://www.imdb.com/title/(tt[0-9]+)/?" with the full url you're expecting.

Also I agree that turning on sorted="yes" would vastly improve the default IMDB scraper's results. IMDB's search is doing far more intelligent sorting than XBMC's basic string comparison.

I've said this about 100 times.

Change the imdb scraper to return ALL titles that the search brings, rather than just the links. The problem is that it's not returning the AKA title names that the search page gives.

eg Infernal Affairs

Should include both "The Departed" and "Infernal Affairs" results, both linking to the same movie.

That way the fuzzy matching in XBMC will work perfectly, _enhancing_ the IMDb results for those cases where it doesn't work well.

Can you find any other cases where the IMDb search would then require sorted="yes" ?

Cheers,
Jonathan

Hi, What changes to I need to make to get IMDB to return all titles?

I am having this issue as its not returning the AKA titles.

Cheers
Si

You need to modify the scraper XML file. This involves parsing the results IMDb gives using regular expressions to generate a set of XML results that XBMC then uses.

Cheers,
Jonathan

So I did some quick tests using my Anime collection. Here are the 13 movies I used for testing:

My Bing+IMDB scraper gets all 13/13 correct.

The default IMDB scraper (unsorted) gets only 7/13. The ones it gets wrong:

Using the default IMDB scrapper and adding the sorted="yes" tag gets 12/13. It can't find this one at all (doesn't even get a wrong hit):

jmarshall: why bother adding a bunch more code to scrape aka names from IMDB results. Is there any case where XBMC's re-order of names provides better results than IMDB's sort order?

Because IMDb may hit the "popular" result first, even though the actual result may be further down?