MovieMeter.nl (Dutch Movies) Scraper development...

Bigfoot87 · 2009-12-18 00:04

The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper.

I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data.

Great news!.. Big Grin

brulsmurf · 2009-12-18 16:22

The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper.

I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data.

Good luck with that! I hope it works out, in which case I will definitely use it. Please note however that you may run into trouble, because the API only allows so many calls from the same IP address. If you access it through a website, all people that use your scraper will access the Moviemeter site through the same IP address (namely the website server's). I think you should check with the Moviemeter API administrator just how many calls will be allowed to see if there is a practical limitation here. I think there will be.

frankied · 2009-12-25 23:05

The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper.

I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data.

Since a couple of days I have a similar setup running. Please find the scraper file attached. It gets movie information and thumbs from moviemeter and adds fanart from themoviedb.

Code:

<?xml version="1.0" encoding="utf-8" standalone="no"?><scraper framework="1.1" date="2009-12-13" name="MovieMeter" content="movies" thumb="moviemeter.jpg" language="nl">

        <CreateSearchUrl clearbuffers="no" dest="3">

                <RegExp input="$$1" output="&lt;url&gt;http://127.0.0.1/moviemeter/search.php?film=\1&lt;/url&gt;" dest="3">

                        <expression></expression>

                </RegExp>

        </CreateSearchUrl>

        <GetSearchResults clearbuffers="no" dest="8">

                <RegExp input="$$5" output="&lt;?xml version=&quot;1.0&quot; encoding=&quot;iso-8859-1&quot; standalone=&quot;yes&quot;?&gt;&lt;results&gt;\1&lt;/results&gt;" dest="8">

                        <RegExp input="$$1" output="&lt;entity&gt;&lt;title&gt;\3 \5&lt;/title&gt;&lt;url&gt;http://127.0.0.1/moviemeter/film.php?filmid=\1&lt;/url&gt;&lt;/entity&gt;" dest="5">

                                <expression repeat="yes">\["filmId"\]=&gt;\n    string\([0-9]*\) "([0-9]*)"\n    \["url"\]=&gt;\n    string\([0-9]*\) "(.*?)"\n    \["title"\]=&gt;\n    string\([0-9]*\) "(.*?)"\n    \["alternative_title"\]=&gt;\n    string\([0-9]*\) "(.*?)"\n    \["year"\]=&gt;\n    string\([0-9]*\) "([0-9]*)"\n    \["average"\]=&gt;</expression>

                        </RegExp>

                        <expression noclean="1"></expression>

                </RegExp>

        </GetSearchResults<GetDetails clearbuffers="no" dest="3">

                <RegExp input="$$8" output="&lt;details&gt;\1&lt;/details&gt;" dest="3">

                        <RegExp input="$$1" output="&lt;title&gt;\3&lt;/title&gt;&lt;year&gt;\4&lt;/year&gt;&lt;thumb&gt;http://www.moviemeter.nl/images/covers/\2&lt;/thumb&gt;&lt;plot&gt;\6&lt;/plot&gt;&lt;rating max=&quot;5&quot;&gt;\9&lt;/rating&gt;" dest="8">

                                <expression noclean="1">search results:array\([0-9]*\) {\n  \["url"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["thumbnail"\]=&gt;\n  string\([0-9]*\) "http://www.moviemeter.nl/images/covers/thumbs/(.*?)"\n  \["title"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["alternative_titles"\]=&gt;\n  .*?\n  \["year"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"\n  \["imdb"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"\n  \["plot"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["mini_series"\]=&gt;\n  string\(1\) "([0-1])"\n  .*?\n  \["actors_text"\]=&gt;\n  string\([0-9]*\) ".*?"\n  \["directors"\]=&gt;\n  .*?\n  \["directors_text"\]=&gt;\n  string\([0-9]*\) ".*?"\n  \["countries"\]=&gt;\n  .*?\n  \["countries_text"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["genres"\]=&gt;\n  .*?\n  \["genres_text"\]=&gt;\n  string\([0-9]*\) ".*?"\n  \["dates_cinema"\]=&gt;\n  .*?\n  \["average"\]=&gt;\n  string\([0-9]*\) "([0-9.]*)"\n  \["votes_count"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"\n  \["filmId"\]=&gt;\n  int\(([0-9]*)\)\n</expression>

                        </RegExp>

                        <RegExp input="$$7" output="&lt;actor&gt;&lt;name&gt;\1&lt;/name&gt;&lt;role&gt;&lt;/role&gt;&lt;/actor&gt;" dest="8+">

                                <RegExp input="$$1" output="\1" dest="7">

                                        <expression noclean="1">\["actors"\]=&gt;\n  array\([0-9]*\) {(.*?)}\n  \["actors_text"\]=&gt;</expression>

                                </RegExp>

                                <expression repeat="yes" noclean="1">\["name"\]=&gt;\n      string\([0-9]*\) "(.*?)"\n      \["voice"\]=&gt;</expression>

                        </RegExp>

                        <RegExp input="$$7" output="&lt;director&gt;\1&lt;/director&gt;" dest="8+">

                                <RegExp input="$$1" output="\1" dest="7">

                                        <expression noclean="1">\["directors"\]=&gt;\n  array\([0-9]*\) {(.*?)}\n  \["directors_text"\]=&gt;</expression>

                                </RegExp>

                                <expression repeat="yes" noclean="1">\["name"\]=&gt;\n      string\([0-9]*\) "(.*?)"\n    }\n</expression>

                        </RegExp>

                        <RegExp input="$$7" output="&lt;genre&gt;\1&lt;/genre&gt;" dest="8+">

                                <RegExp input="$$1" output="\1" dest="7">

                                        <expression noclean="1">\["genres"\]=&gt;\n  array\([0-9]*\) {(.*?) }</expression>

                                </RegExp>

                                <expression repeat="yes" noclean="1">\[[0-9]*\]=&gt;\n    string\([0-9]*\) "(.*?)"\n</expression>

                        </RegExp>

                        <RegExp input="$$1" output="&lt;url function=&quot;GetTMDBId&quot;&gt;http://api.themoviedb.org/2.0/Movie.imdbLookup?imdb_id=tt\1&amp;api_key=57983e31fb435df4df77afb854740ea9&lt;/url&gt;" dest="8+">

                                <expression noclean="1">\["imdb"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"</expression>

                        </RegExp>

                        <expression noclean="1"></expression>

                </RegExp>

        </GetDetails><GetTMDBId dest="8">

                <RegExp input="$$1" output="&lt;details&gt;&lt;url function=&quot;GetTMDBFanart&quot;&gt;http://api.themoviedb.org/2.0/Movie.getInfo?id=\1&amp;amp;api_key=57983e31fb435df4df77afb854740ea9&lt;/url&gt;&lt;/details&gt;" dest="8">

                        <expression>&lt;id&gt;([0-9]*)&lt;/id&gt;</expression>

                </RegExp>

        </GetTMDBId>

        <GetTMDBFanart dest="8">

                <RegExp input="$$2" output="&lt;details&gt;&lt;fanart url=&quot;http://themoviedb.org/image/backdrops&quot;&gt;\1&lt;/fanart&gt;&lt;/details&gt;" dest="8">

                        <RegExp input="$$1" output="&lt;thumb preview=&quot;/\1/\2_poster.\3&quot;&gt;/\1/\2.\3&lt;/thumb&gt;" dest="2">

                                <expression repeat="yes">&lt;backdrop size="original"&gt;.*?/([0-9]+)/([^\.]+)\.([^&lt;]*)&lt;/backdrop&gt;</expression>

                        </RegExp>

                        <expression noclean="1">(.+)</expression>

                </RegExp>

        </GetTMDBFanart>

</scraper>

redje · 2009-12-26 12:53

I was thrilled to finally see a dutch movie scraper, my gf will be very pleased.
i tried you're setting in a file moviemeter.xml, but the scraper doesn't come up in the scraper selection menu. Any idea what i'm doing wrong?

jpschouten · 2009-12-26 16:01

Same as redje ! very exited to see a moviemeter scraper workin on !! but how to use this ? Also my GF will be very pleased i think MANY MANY GF and BF will be pleased

frankied · 2009-12-26 16:27

redje Wrote:I was thrilled to finally see a dutch movie scraper, my gf will be very pleased.
i tried you're setting in a file moviemeter.xml, but the scraper doesn't come up in the scraper selection menu. Any idea what i'm doing wrong?

jpschouten Wrote:Same as redje ! very exited to see a moviemeter scraper workin on !! but how to use this ? Also my GF will be very pleased i think MANY MANY GF and BF will be pleased

As soon as I find a way to include attachments I will post instructions to get a moviemeter-based scraper up and running .....

jpschouten · 2009-12-26 16:35

maybe you can upload it to http://www.megaupload.com/ or something.
And supply us with the link and instructions !?

frankied · 2009-12-26 22:54

jpschouten Wrote:maybe you can upload it to http://www.megaupload.com/ or something.
And supply us with the link and instructions !?

Step 1: Get a key to access the moviemeter API from http://www.moviemeter.nl/site/registerclient/

Step 2: Setup a local website which gives access to the moviemeter API

sudo apt-get install nanoweb
create a directory moviemeter in /usr/share/nanoweb/defaultroot/
put the contents from moviemeter API in the moviemeter directory. Replace the xxxx with your own key in de search.php and film.php files.

Step 3: Put the scraper files from Moviemeter scraper to /usr/share/xbmc/system/scrapers/video/

This should be it ...

If someone could host the ¨fake¨ moviemeter website that gives access to the API, it can become a regular scraper without having to install a local webserver.

jpschouten · 2009-12-27 10:39

Strange, i followed your guide. Can select Moviemeter at the scraper-section. But when i try to update the library nothing happends. He doesn't get info ? I checked permissions on the files, any idea Confused

frankied · 2009-12-27 11:49

jpschouten Wrote:Strange, i followed your guide. Can select Moviemeter at the scraper-section. But when i try to update the library nothing happends. He doesn't get info ? I checked permissions on the files, any idea

First check if the local website is up and running from your browser: http://127.0.0.1/moviemeter/search.php?film=madagascar

You should get a page like this:

search results:array(2) { [0]=> array(12) { ["filmId"]=> string(5) "13894" ["url"]=> string(35) "http://www.moviemeter.nl/film/13894" ["title"]=> string(10) "Madagascar" ["alternative_title"]=> string(0) "" ["year"]=> string(4) "2005" ["average"]=> string(4) "3.19" ["votes_count"]=> string(4) "2072" ["similarity"]=> string(6) "100.00" ["directors_text"]=> string(45) "geregisseerd door Eric Darnell en Tom McGrath" ["actors_text"]=> string(64) "met de stemmen van Ben Stiller, Chris Rock en Jada Pinkett Smith" ["genres_text"]=> string(19) "Animatie / Avontuur" ["duration"]=> string(2) "80" } [1]=> array(12) { ["filmId"]=> string(5) "46166" ["url"]=> string(35) "http://www.moviemeter.nl/film/46166" ["title"]=> string(27) "Madagascar: Escape 2 Africa" ["alternative_title"]=> string(0) "" ["year"]=> string(4) "2008" ["average"]=> string(4) "3.22" ["votes_count"]=> string(3) "948" ["similarity"]=> string(5) "54.05" ["directors_text"]=> string(45) "geregisseerd door Eric Darnell en Tom McGrath" ["actors_text"]=> string(61) "met de stemmen van Ben Stiller, Chris Rock en David Schwimmer" ["genres_text"]=> string(19) "Animatie / Avontuur" ["duration"]=> string(2) "89" } }

Bigfoot87 Senior Member Posts: 134 Joined: Dec 2006 Reputation: 0 Location: Netherlands	2009-12-18 00:04 Post: #61 The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper. I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data. Great news!..
	find quote

redje Member Posts: 93 Joined: Jan 2009 Reputation: 0	2009-12-26 12:53 Post: #64 I was thrilled to finally see a dutch movie scraper, my gf will be very pleased. i tried you're setting in a file moviemeter.xml, but the scraper doesn't come up in the scraper selection menu. Any idea what i'm doing wrong?
	find quote

jpschouten Junior Member Posts: 17 Joined: Oct 2009 Reputation: 0	2009-12-26 16:01 Post: #65 Same as redje ! very exited to see a moviemeter scraper workin on !! but how to use this ? Also my GF will be very pleased i think MANY MANY GF and BF will be pleased
	find quote

jpschouten Junior Member Posts: 17 Joined: Oct 2009 Reputation: 0	2009-12-26 16:35 Post: #67 maybe you can upload it to http://www.megaupload.com/ or something. And supply us with the link and instructions !?
	find quote

jpschouten Junior Member Posts: 17 Joined: Oct 2009 Reputation: 0	2009-12-27 10:39 Post: #69 Strange, i followed your guide. Can select Moviemeter at the scraper-section. But when i try to update the library nothing happends. He doesn't get info ? I checked permissions on the files, any idea
	find quote