• 1
  • 3
  • 4
  • 5(current)
  • 6
  • 7
  • 30
MovieMeter.nl (Dutch Movies) Scraper development...
#61
The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper.

I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data.

Great news!.. Big Grin
Reply
#62
The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper.

I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data.

Good luck with that! I hope it works out, in which case I will definitely use it. Please note however that you may run into trouble, because the API only allows so many calls from the same IP address. If you access it through a website, all people that use your scraper will access the Moviemeter site through the same IP address (namely the website server's). I think you should check with the Moviemeter API administrator just how many calls will be allowed to see if there is a practical limitation here. I think there will be.
Reply
#63
The_Ghost16 Wrote:Yesterday i started with a moviemeter scraper.

I'm using a website between the scraper and the api moviemeter provided. So at the moment i'm working on the output and when i'm finished testing with the scraper tool i will install it on my own machines (windows pc and xbox) and see how it will work with real data.

Since a couple of days I have a similar setup running. Please find the scraper file attached. It gets movie information and thumbs from moviemeter and adds fanart from themoviedb.

Code:
<?xml version="1.0" encoding="utf-8" standalone="no"?><scraper framework="1.1" date="2009-12-13" name="MovieMeter" content="movies" thumb="moviemeter.jpg" language="nl">
        <CreateSearchUrl clearbuffers="no" dest="3">
                <RegExp input="$$1" output="&lt;url&gt;http://127.0.0.1/moviemeter/search.php?film=\1&lt;/url&gt;" dest="3">
                        <expression></expression>
                </RegExp>
        </CreateSearchUrl>
        <GetSearchResults clearbuffers="no" dest="8">
                <RegExp input="$$5" output="&lt;?xml version=&quot;1.0&quot; encoding=&quot;iso-8859-1&quot; standalone=&quot;yes&quot;?&gt;&lt;results&gt;\1&lt;/results&gt;" dest="8">
                        <RegExp input="$$1" output="&lt;entity&gt;&lt;title&gt;\3 \5&lt;/title&gt;&lt;url&gt;http://127.0.0.1/moviemeter/film.php?filmid=\1&lt;/url&gt;&lt;/entity&gt;" dest="5">
                                <expression repeat="yes">\["filmId"\]=&gt;\n    string\([0-9]*\) "([0-9]*)"\n    \["url"\]=&gt;\n    string\([0-9]*\) "(.*?)"\n    \["title"\]=&gt;\n    string\([0-9]*\) "(.*?)"\n    \["alternative_title"\]=&gt;\n    string\([0-9]*\) "(.*?)"\n    \["year"\]=&gt;\n    string\([0-9]*\) "([0-9]*)"\n    \["average"\]=&gt;</expression>
                        </RegExp>
                        <expression noclean="1"></expression>
                </RegExp>
        </GetSearchResults<GetDetails clearbuffers="no" dest="3">
                <RegExp input="$$8" output="&lt;details&gt;\1&lt;/details&gt;" dest="3">
                        <RegExp input="$$1" output="&lt;title&gt;\3&lt;/title&gt;&lt;year&gt;\4&lt;/year&gt;&lt;thumb&gt;http://www.moviemeter.nl/images/covers/\2&lt;/thumb&gt;&lt;plot&gt;\6&lt;/plot&gt;&lt;rating max=&quot;5&quot;&gt;\9&lt;/rating&gt;" dest="8">
                                <expression noclean="1">search results:array\([0-9]*\) {\n  \["url"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["thumbnail"\]=&gt;\n  string\([0-9]*\) "http://www.moviemeter.nl/images/covers/thumbs/(.*?)"\n  \["title"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["alternative_titles"\]=&gt;\n  .*?\n  \["year"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"\n  \["imdb"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"\n  \["plot"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["mini_series"\]=&gt;\n  string\(1\) "([0-1])"\n  .*?\n  \["actors_text"\]=&gt;\n  string\([0-9]*\) ".*?"\n  \["directors"\]=&gt;\n  .*?\n  \["directors_text"\]=&gt;\n  string\([0-9]*\) ".*?"\n  \["countries"\]=&gt;\n  .*?\n  \["countries_text"\]=&gt;\n  string\([0-9]*\) "(.*?)"\n  \["genres"\]=&gt;\n  .*?\n  \["genres_text"\]=&gt;\n  string\([0-9]*\) ".*?"\n  \["dates_cinema"\]=&gt;\n  .*?\n  \["average"\]=&gt;\n  string\([0-9]*\) "([0-9.]*)"\n  \["votes_count"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"\n  \["filmId"\]=&gt;\n  int\(([0-9]*)\)\n</expression>
                        </RegExp>
                        <RegExp input="$$7" output="&lt;actor&gt;&lt;name&gt;\1&lt;/name&gt;&lt;role&gt;&lt;/role&gt;&lt;/actor&gt;" dest="8+">
                                <RegExp input="$$1" output="\1" dest="7">
                                        <expression noclean="1">\["actors"\]=&gt;\n  array\([0-9]*\) {(.*?)}\n  \["actors_text"\]=&gt;</expression>
                                </RegExp>
                                <expression repeat="yes" noclean="1">\["name"\]=&gt;\n      string\([0-9]*\) "(.*?)"\n      \["voice"\]=&gt;</expression>
                        </RegExp>
                        <RegExp input="$$7" output="&lt;director&gt;\1&lt;/director&gt;" dest="8+">
                                <RegExp input="$$1" output="\1" dest="7">
                                        <expression noclean="1">\["directors"\]=&gt;\n  array\([0-9]*\) {(.*?)}\n  \["directors_text"\]=&gt;</expression>
                                </RegExp>
                                <expression repeat="yes" noclean="1">\["name"\]=&gt;\n      string\([0-9]*\) "(.*?)"\n    }\n</expression>
                        </RegExp>
                        <RegExp input="$$7" output="&lt;genre&gt;\1&lt;/genre&gt;" dest="8+">
                                <RegExp input="$$1" output="\1" dest="7">
                                        <expression noclean="1">\["genres"\]=&gt;\n  array\([0-9]*\) {(.*?) }</expression>
                                </RegExp>
                                <expression repeat="yes" noclean="1">\[[0-9]*\]=&gt;\n    string\([0-9]*\) "(.*?)"\n</expression>
                        </RegExp>
                        <RegExp input="$$1" output="&lt;url function=&quot;GetTMDBId&quot;&gt;http://api.themoviedb.org/2.0/Movie.imdbLookup?imdb_id=tt\1&amp;api_key=57983e31fb435df4df77afb854740ea9&lt;/url&gt;" dest="8+">
                                <expression noclean="1">\["imdb"\]=&gt;\n  string\([0-9]*\) "([0-9]*)"</expression>
                        </RegExp>
                        <expression noclean="1"></expression>
                </RegExp>
        </GetDetails><GetTMDBId dest="8">
                <RegExp input="$$1" output="&lt;details&gt;&lt;url function=&quot;GetTMDBFanart&quot;&gt;http://api.themoviedb.org/2.0/Movie.getInfo?id=\1&amp;amp;api_key=57983e31fb435df4df77afb854740ea9&lt;/url&gt;&lt;/details&gt;" dest="8">
                        <expression>&lt;id&gt;([0-9]*)&lt;/id&gt;</expression>
                </RegExp>
        </GetTMDBId>
        <GetTMDBFanart dest="8">
                <RegExp input="$$2" output="&lt;details&gt;&lt;fanart url=&quot;http://themoviedb.org/image/backdrops&quot;&gt;\1&lt;/fanart&gt;&lt;/details&gt;" dest="8">
                        <RegExp input="$$1" output="&lt;thumb preview=&quot;/\1/\2_poster.\3&quot;&gt;/\1/\2.\3&lt;/thumb&gt;" dest="2">
                                <expression repeat="yes">&lt;backdrop size="original"&gt;.*?/([0-9]+)/([^\.]+)\.([^&lt;]*)&lt;/backdrop&gt;</expression>
                        </RegExp>
                        <expression noclean="1">(.+)</expression>
                </RegExp>
        </GetTMDBFanart>
</scraper>
Reply
#64
I was thrilled to finally see a dutch movie scraper, my gf will be very pleased.
i tried you're setting in a file moviemeter.xml, but the scraper doesn't come up in the scraper selection menu. Any idea what i'm doing wrong?
Reply
#65
Same as redje ! very exited to see a moviemeter scraper workin on !! but how to use this ? Also my GF will be very pleased i think MANY MANY GF and BF will be pleased
Reply
#66
redje Wrote:I was thrilled to finally see a dutch movie scraper, my gf will be very pleased.
i tried you're setting in a file moviemeter.xml, but the scraper doesn't come up in the scraper selection menu. Any idea what i'm doing wrong?

jpschouten Wrote:Same as redje ! very exited to see a moviemeter scraper workin on !! but how to use this ? Also my GF will be very pleased i think MANY MANY GF and BF will be pleased

As soon as I find a way to include attachments I will post instructions to get a moviemeter-based scraper up and running .....
Reply
#67
maybe you can upload it to http://www.megaupload.com/ or something.
And supply us with the link and instructions !?
Reply
#68
jpschouten Wrote:maybe you can upload it to http://www.megaupload.com/ or something.
And supply us with the link and instructions !?

Step 1: Get a key to access the moviemeter API from http://www.moviemeter.nl/site/registerclient/

Step 2: Setup a local website which gives access to the moviemeter API

sudo apt-get install nanoweb
create a directory moviemeter in /usr/share/nanoweb/defaultroot/
put the contents from moviemeter API in the moviemeter directory. Replace the xxxx with your own key in de search.php and film.php files.

Step 3: Put the scraper files from Moviemeter scraper to /usr/share/xbmc/system/scrapers/video/

This should be it ...

If someone could host the ¨fake¨ moviemeter website that gives access to the API, it can become a regular scraper without having to install a local webserver.
Reply
#69
Strange, i followed your guide. Can select Moviemeter at the scraper-section. But when i try to update the library nothing happends. He doesn't get info ? I checked permissions on the files, any idea Huh
Reply
#70
jpschouten Wrote:Strange, i followed your guide. Can select Moviemeter at the scraper-section. But when i try to update the library nothing happends. He doesn't get info ? I checked permissions on the files, any idea Huh

First check if the local website is up and running from your browser: http://127.0.0.1/moviemeter/search.php?film=madagascar

You should get a page like this:

search results:array(2) { [0]=> array(12) { ["filmId"]=> string(5) "13894" ["url"]=> string(35) "http://www.moviemeter.nl/film/13894" ["title"]=> string(10) "Madagascar" ["alternative_title"]=> string(0) "" ["year"]=> string(4) "2005" ["average"]=> string(4) "3.19" ["votes_count"]=> string(4) "2072" ["similarity"]=> string(6) "100.00" ["directors_text"]=> string(45) "geregisseerd door Eric Darnell en Tom McGrath" ["actors_text"]=> string(64) "met de stemmen van Ben Stiller, Chris Rock en Jada Pinkett Smith" ["genres_text"]=> string(19) "Animatie / Avontuur" ["duration"]=> string(2) "80" } [1]=> array(12) { ["filmId"]=> string(5) "46166" ["url"]=> string(35) "http://www.moviemeter.nl/film/46166" ["title"]=> string(27) "Madagascar: Escape 2 Africa" ["alternative_title"]=> string(0) "" ["year"]=> string(4) "2008" ["average"]=> string(4) "3.22" ["votes_count"]=> string(3) "948" ["similarity"]=> string(5) "54.05" ["directors_text"]=> string(45) "geregisseerd door Eric Darnell en Tom McGrath" ["actors_text"]=> string(61) "met de stemmen van Ben Stiller, Chris Rock en David Schwimmer" ["genres_text"]=> string(19) "Animatie / Avontuur" ["duration"]=> string(2) "89" } }
Reply
#71
Hi frankied,

I managed to get your scraper working, but all films starting with "the" don't get scraped. Also none of my films gets the runtime scraped and on my main movie view none of my movies has the plot, it is in the movie information page. I think this is because XBMC uses plot and summary.

But at last there's a possibility to use moviemeter, which is fantastic! Thank you very much!
Reply
#72
If you follow the Guide the search.php file is in the root of the webserver, so what I did is edit the moviemeter.xml file to reflect these changed (so remove /moviemeter on 2 places)

Or you can add the /moviemeter folder in the /usr/share/nanoweb/defaultroot/ folder, and move the files overthere.

Frankie first of all Great work!!

I have two small issues. The plot shows up as something like "hidden because of spoilers" and I miss the length/duration of the movie.

bye, Joep
Reply
#73
Oke i managed to get it to work. My own fault of course, My girlfriend is very happy with it and me to it works very good Wink thx !!!!! you r a hero of many dutch dude(dets)
Reply
#74
did you guys managed to get it running under windows xp or vista? i've tried some webserver software, but i can't get it running. Can someone point me in the right direction with windows?
Reply
#75
Hi Redje,

You need a PHP capable webserver. Depending on where you put the files
[HTML]
http://127.0.0.1/moviemeter/search.php?film=madagascar[/HTML]
or
[HTML]http://127.0.0.1/search.php?film=madagascar[/HTML]

Should give you the output like in post 70

If you get that, your webserver is running ok.

In the scaper file is a link to these locations, make sure they match (I removed the /moviemeter part)
Reply
  • 1
  • 3
  • 4
  • 5(current)
  • 6
  • 7
  • 30

Logout Mark Read Team Forum Stats Members Help
MovieMeter.nl (Dutch Movies) Scraper development...5