German IMDB scraper, please test it and give feedback - Printable Version
+- XBMC Community Forum (http://forum.xbmc.org)
+-- Forum: Help and Support (/forumdisplay.php?fid=33)
+--- Forum: Add-ons Help and Support (/forumdisplay.php?fid=27)
+---- Forum: Metadata scrapers (/forumdisplay.php?fid=147)
+---- Thread: German IMDB scraper, please test it and give feedback (/showthread.php?tid=75121)
German IMDB scraper, please test it and give feedback - Eisbahn - 2010-06-04 23:33
new version online: http://github.com/Eisbahn/IMDb_de-Scraper/zipball/3.0.5
v3.0.4 and v2.0.2 out now: <http://github.com/Eisbahn/IMDb_de-Scraper/>
just finished a first version after some work: german scraper for IMDB (in german language). Actual Version 1.0.0 is on http://ul.to/v5d9j0 ready for test. It grabs every tag availabel, only Trailer is not implemented (because for me it's a useless feature). Please feel free to report bugs or issues,
latest Version 2.0.0 for XBMC v9.11 can be found here:
What is _not_ working
<premiered>Premierendatum</premiered> not im-/exported to XBMC
<aired>???</aired> only for TV-Shows/series?
<set>???</set> don't know what this is
<artist>???</artist> difference to actor?
<status>???</status> don't know what this is
<certification>Altersfreigabe für alle Staaten außer D</certification> not im-/exported to XBMC
<sorttitle>alternative Filmtitel</sorttitle>only first titel is im-/exported to XBMC
<code>???</code> don't know what this is, I think it's the codec => no sense to import anything in this field
<trailer>Trailer</trailer> senseless for me because the hole DVD is in XBMC present
Any hints for the corrupted tags are highly welcome.
- Spaggi - 2010-06-05 00:35
Great work! Will test on the weekend
- Eisbahn - 2010-06-05 09:27
Because I'm new to XBMC: what tags are supported/should be provided by a scraper?
Could you give me an overview of mandantory and optional tags?
- vdrfan - 2010-06-05 10:55
Eisbahn Wrote:Because I'm new to XBMC: what tags are supported/should be provided by a scraper?
Check out the other scrapers. The imdb.com is pretty feature complete.
- donabi - 2010-06-05 10:56
well, that differs very much.
some users "need" studio-tags, to have fancy icons in the skin.
or the narator.
others, like me, just need things like playtime, year, actors, fsk (mpaa) and ONE genre.
the orignal imdb-scraper gets a lot of genre-tags.
which makes the genre-filter sense-less.
we would like to see you at german xbmc.de
- Eisbahn - 2010-06-05 18:20
Hmmm, sorry. Do we have a spec showing which tags are mandantory/optional? If not: how can I figure out which tags are supported? The IMDB com scraper fetches no infos about sound, subtitle, video-format (if I looked right), in several screenshots I could see infos about these things... So the answer: please do reverse engineering because everybody can implement tags however he/she likes is a bit contra productive and shows kind of quick-and-dirty-hacking without any concept? Is this the XBMC style?
@donabi: to cut some infos away is not a real problem and done in few seconds. But gathering all possible things is a bit more complicated. So first I would have a scraper which gets all infos.
If you have a decription of the alowed tags, please provide it. Is the order/sequence relevant, what tags are supported, what format is expected and so on. If the german board has active members, why not. But to be honest: think after the scraper my active work is over :=(
@all: Where can I get infos which tags are supported by XBMC? If the skins shows the infos doesn't matter at all, think a "good scrapper" should gather as much as possible. For the result of a scraper: is the order/sequence relevant, what tags are supported, what format is expected and so on. Today all I've done is reverse engineering, but I think thats not the right way...
- olympia - 2010-06-05 21:31
As for the starting point:
I think you are chasing something like the first result?
Other than that, I am not sure you are seriously calling "reverse engineering" to just have a look at what tags are being used by other scrapers.
- Eisbahn - 2010-06-05 23:04
great, I found google. If you know the right words and do not type "scraper, tags, xbmc" as a newbee or anything like that, it realy works. If my questions are so easy: why do I get only from you an answer? Think it's a bit frustrating for both of us: for you as expert and me as new user...
- The set tag is for a standalone XBMC useless because you could not edit the tag before importing, so should not be used by a scraper. Am I right?
- what about the order. Is it relevant? Seemed to be not (looking at your nfo and the imdb.com output)
- fileinfos are imported by XBMC by analysing the video file on its own without interaction?
 new version:
- year gets imported if quartal is added, like in "Insomnia (2002)"
- importing up to 6 genres (9 easy possible)
- triming of spaces
=> to come: all tags like in the nfo
- Nicezia - 2010-06-05 23:15
All tags are optional , but i would say its best that the TITLE is at least supplied
of course it goes without saying that actor, audio (inside stream info), video(inside stream info) and subtitle(inside stream info) are multiple instance and optional
- olympia - 2010-06-06 08:57
Eisbahn Wrote:- The set tag is for a standalone XBMC useless because you could not edit the tag before importing, so should not be used by a scraper. Am I right?Yes, it's only useful when you have an xbmc compliant external nfo to import from. Nevertheless you couldn't even scrape this info from anywhere
Eisbahn Wrote:- what about the order. Is it relevant? Seemed to be not (looking at your nfo and the imdb.com output)Order doesn't matter
Eisbahn Wrote:- fileinfos are imported by XBMC by analysing the video file on its own without interaction?Yes, if this option is enabled in xbmc. But obviously this is again an info what you couldn't scrape from a web site. These tags are existing for an nfo, because you might don't want xbmc to do the extraction from the media file in itself, because you use an external nfo manager for that purposes and you want xbmc to import the data generated by that.
- Nicezia - 2010-06-06 09:11
olympia Wrote:Yes, it's only useful when you have an xbmc compliant external nfo to import from. Nevertheless you couldn't even scrape this info from anywhere
i don't mean to be correcting you here, but i have written a scraper for AEBN that imports the Movie Series name as set, and XBMC parses it from the scraper just fine, so set can as well be used with scraping
Any tag that can be exported from XBMC (do an export on a small library for an example) can be imported by XBMC from scraper or nfo....
however, you are very right in that its best to let XBMC or an external nfo manager to handle fileinfo
- Nicezia - 2010-06-06 09:24
Eisbahn Wrote:- importing up to 6 genres (9 easy possible)
if you used a sepearating nested RegExp, you could import infinite amount of genres
for instance say something is formated as such
First copy the part that you need to a buffer (8 in this case)
and then parent that with a regular expression that repeatedly finds the comma seperated genres (getting the input from the string you copied to $$8)
so that the end product is something like this
- vdrfan - 2010-06-06 11:39
Thanks for the good information Nicezia! Note, as of SVN revision r30825 the way we handle the runtime/duration has slightly changed.
The following VideoInfoTags (streamdetails) are exported from the actual media file: codec, aspect, width, height and duration (new).
In order to make sure the runtime is shown properly, make sure the scraper only returns minutes (numeric) only. This value is used in case the meta extraction is disabled and/or we somehow fail to extract it.
- Nicezia - 2010-06-06 12:01
vdrfan Wrote:Thanks for the good information Nicezia! Note, as of SVN revision r30825 the way we handle the runtime/duration has slightly changed.
Noted, and i will adjust ScraperXML code accordingly
@vdrfan, also noticed there are a few other tags not mentioned anywhere else (country, sorttitle, epbookmark, originaltitle) and that premiered though taken from the nfo/scraper, doesn't seem to store into database at all (at least in the last version i'm basing off of which is before the add-on merge, and therefore when importing the file this info is lost, if its even provided)
are these extra tags depreciated tags that haven't been removed from code or added tags (only just now getting to a point where i can read C++ code as well as CSharp) and is the Premiered getting lost fromthe database an oversight?
- Eisbahn - 2010-06-06 23:59
sorry for late reply, but we had a wonderful day, relaxing with my wife and kids. But i've done a little bit of work and some new questions:
What is the content of the tag <outline>? Some more infos as tagline but not as much as in plot or plotsummary? Couldn't find it... At the moment I put in this tag a "short" plot (the one given at the main overview of IMDB).
What about <certification>? Is it deprecated and only MPAA is used instead?
Because of different DVDs, I've got more than one mpaa tag, e.g. 12years heavy cut, 16years cut, 18years uncut (it's not a single instance) at "The Rock" (IMDB-ID = tt0117500)
Is <originaltitle> a subset of <sorttitle>, e.g.
What about the function GetIMDBThumbs? Does it fetch all pics from IMDB, or only the posters (and maybe product)? What are the constants SX, SY, SX$INFO and SY$INFO (or what is this)? Why is the function not repeated (think the users wants more than one thumbnail)? Don't know exactly what this function should do. Pointing to <http://www.imdb.de/title/tt0499549/mediaindex?refine=poster>? Any help?
How can I call a site without getting a "&" to "&" cleaned? Actually I used a function which removes the & and makes an & into the links :=( The "no HTML clean" tag does not work at all...
ok = meaning my scraper gathers the corresponding infos
n/a = not for import use or no infos given on german imdb site
stc = still to come => maybe implemented in future release (meaning: think it's a useless feature...)
What format should <premiered> have? String with month written out, or date?