[MOD] Universal Scraper+ v2.1.6
#1
I was finally able to solve the problems I encountered and finish this mod.

So, what does this mod do?
It provides a couple of interesting (I hope Wink) modifications to the wonderful Universal Scraper:

- Fixed a problem that didn't store score if tomatometer was 0%
In some cases, if tomatometer was 0%, the score wasn't correctly stored. Now it is (not sure, but it may show 0.1 in some cases)
- Added possibility to choose original title language
This has been added for both imdb and tmbd. Look for the option in tab general.
- Added audience tomatometer and average scores
New option for rating scraping from rotten tomatoes. Look for the option in rating, option "Use the score aggregated from".
- Added possibility of encoding of icons and audience percentage in votes
Possibility to use the votes field to store information about audience tomatometer/average, rotten/fresh/certified fresh icon and popcorn/spilled icon. See later for details. This option is only available if "Use the score aggregated from" is set to top critics or all critics. Option not set by default.
- Added possibility of google search for getting rotten tomatoes page
Great improvement (at least from my tests) in correct rotten tomatoes page retrieval by using google search instead of sometimes incorrect imdb id alias. Look for the option in rating tab, if rating source is set to rotten tomatoes. I've done some tests and it behaves superbly, it correctly solves known titles with a corrupt rotten tomatoes - imdb binding (i robot, the incredibles...). The only failure I've found so far is with "Introducing the Dwights", just because that movie is also known as Clubland and google provides as first result with the movie from 2000. For that case I found out that temporarily disabling google search in options, rescrape the single movie, and re-enabling google search does the trick.



FAQ:

What do you mean by "encoding of icons and audience percentage in votes"?
Let me start by explaining why would I think of such a crazy thing. I love rotten tomatoes ratings but, unfortunately, xbmc offers only 1 field for storing ratings. Now, critics sometimes don't really reflect how audience perceive a movie, and the other way around also applies. So I tried to think of a way to store both values in xbmc.
The number of votes is a special field that I don't consider very often, so I thought: "why not to try and store what you want in there?"
I searched the documentation a bit and, unfortunately, I've seen that it's not possible to use this field as is in a skin, but only together with rating. So I thought that, since some work was to be done on the skin part, I could store even more information in that field. For the time being I've come up with:
- audience score (either tomatometer or average, according to the selection in the options)
- a flag indicating if critics score is rotten, fresh or certified fresh
- a flag indicating if audience score is "spilled" or "popcorn"
This information is stored in the following string:
aAAAtTpP
where "a" is fixed (just to be able to recognize it's audience score), "AAA" is the audience score, "t" is fixed, "T" is 0 for rotten, 1 for fresh, 2 for certified fresh, "p" is fixed, "P" is 0 for spilled and 1 for popcorn
Obviously now it's up to skinners to use this information in a useful way. I tried to put them in a way that allows them to be handled, but I'm not an expert in skinning, so I'm open to options.

How does google search work?
It's pretty simple: it queries google for the title, the year and it appends "rottentomatoes.com", just to be sure to let google know we're looking something on rottentomatoes.com. Given the results page, it picks the first results that contains www.rottentomatoes.com/m/[movie here]. From what I've seen, it works pretty well. Of course, nothing is perfect, and I can't control google results, so we have to put some faith into google to get to the right rotten tomatoes page.



Example of skin using this encoding:
Image



INSTALLATION:
I have to be honest, I don't really know how zip file installation works behind the scenes in xbmc so, please, DO NOT INSTALL THE ATTACHED ZIP FILE. Just extract the contents and add the 4 folders to you userdata/addons folder. The userdata folder may be in your XBMC installation folder if you run XBMC in portable mode. If you run it normally, you may still find it, but you must put the 4 folders in:
- Users\%userprofile%\AppData\Roaming\XBMC\addons for Windows Vista/7
- C:\Documents and Settings\%userprofile%\Application Data\XBMC\addons for Windows XP
Make a backup of the folders you're going to overwrite, so you can go back to how things were, should something go wrong =)

Now the link Smile: https://dl.dropbox.com/s/48auu3wm7s4ajcw...6.zip?dl=1



CHANGELOG:
2.1.6:
- Updated files to reflect recent modifications in Universal Scraper (v 2.1.6)
2.1.5:
- Updated files to reflect recent modifications in Universal Scraper (v 2.1.5)
2.1.4:
- Updated files to reflect recent modifications in Universal Scraper (v 2.1.4)
- Updated imdb common library to 2.6.2
2.1.2:
- Updated files to reflect recent modifications in Universal Scraper (v 2.1.2)
- Updated tmdb common library to 2.9
1.9.4:
- Updated files to reflect recent modifications in Universal Scraper (v 1.9.4)
- Updated imdb common library to 2.6.1
1.9.3:
- Updated files to reflect recent modifications in Universal Scraper (v 1.9.3)
1.9:
- Updated files to reflect recent modifications in Universal Scraper (v 1.9 - Interim versions 1.7, 1.8 and 1.8.1 are included in 1.9)
- Updated imdb and tmdb common libraries fixed as well
1.6.2:
- Updated files to reflect recent modifications in Universal Scraper (v 1.6.2)
1.6.1:
- Updated files to reflect recent modifications in Universal Scraper (v 1.6.1)
1.5:
- Updated files to reflect recent modifications in Universal Scraper (v 1.5.0) and TMDb core library
1.3:
- Updated files to reflect recent modifications in Universal Scraper (v 1.3.0)
- Changed version numbering so it matches Universal Scraper's
1.0:
- Initial release




NEW!!!

SKIN (only for Eden, I will start working on a version for Frodo as soon as Frodo bocomes final):
I've been working on a personal modification of the wonderful reFocus BIG in order to integrate Rotten Tomatoes scores at a visual level (see screenshot above for an example). I've finally reached a point where modifications are pretty much how I like them, so I thought I might share that as well.

Instructions:
- Go over to this thread and download the complete skin
- Install the skin
- Download my mod here
- Extract the files and copy them over to the skin folder (overwrite as requested). As usual, make a backup copy of the original files. In my package, I've included reworked sound effects so that they play at a normal level on my machine. This may not be the case on yours, so try them out if you want, and choose those which you feel more comfortable with.
- Launch XBMC, load reFocus BIG, customize the skin to your tastes
- To enable Rotten Tomatoes scores, go in skin settings --> View settings and enable "Show ratings like on Rotten Tomatoes"
- If you scraped your movies with the special encoding enabled (in order to scrape both critics and audience scores), enable "Show also audience using special encoding". This will give you 2 visible scores and the certified fresh icon (if the movie is certified fresh, that is)

Some info:
- Rotten Tomatoes scores are visible in all movie views, as well as in OSDs when playing movies. In all other views (TV series, music...) the usual star system is in place. I can't certify it will work 100% of times, but on my system it works like a charm
- Some views that can have mixed content like "now playing", will not show Rotten Tomatoes scores, even if they include movies. I haven't found a way to show the proper scores only on movies in this case
- When viewing search results, if Rotten Tomatoes scores are active, the score shown is the critics score, in percentage
- There are some other tweaks to the GUI compared to reFocus BIG, pretty much to align it to my tastes
- I don't take credit for the skin, just for the Rotten Tomatoes part. So if you like the skin, go over the proper thread and drop a thank you, or rate it =)



Drop a line if you use it or find a bug! Wink
Reply
#2
I'm trying to implement what I suggested in the official universal scraper thread: getting to the actual rotten tomatoes page using a google search. Say, for example, I'm looking for iron man, my google search link would be:
http://www.google.com/#hl=en&q=iron man+2008+site:rottentomatoes.com

Now, checking it out in a browser is an easy job, and it provides as first result pretty much every time the correct link. The problem is that, when I try to use that link in a custom function in the scraper, what I get back is not the page I see in the browser, but a lot of javascript and almost no html which, I guess, redirects to the actual result page.

Has anyone any experience with google searches through a scraper?

Edit: apparently the actual html code of the page I see in the browser is what I get with the scraper (I was analyzing the page html with opera dragonfly), so I guess that results are returned through ajax...
Reply
#3
Found the problem: I wasn't converting spaces to %20 in url construction. Also, I guess I wasn't escaping : (in site:rottentomatoes.com), so it was somehow failing to provide the correct results (yet not a valid page).

Anyway, full mod released, enjoy and provide feedback! Wink
Reply
#4
Updated to version 1.3 to reflect new modifications in Universal Scraper.
Reply
#5
Sweet! Already updating my library. Any updates on skin development for the icons? Looks very nice!

Keep up the good work!
Reply
#6
Still working on it.
I've just started learning skins, so it's taking a bit longer than I initially expected. But I'm not giving up =)
Reply
#7
Forgive my ignorance but when you say to unzip the files and add them to the userdata/addons folder I can't seem to find that path. I am running XBMC on a Windows 7 64 Bit computer. When I go to C:\Program Files (x86)\XBMC\UserData I only see a folder called PictureIcon and 4 XML docs. I DO have C:\Program Files (x86)\XBMC\addons with a bunch of metadata.xxxxxx.xxxxx in there so I am assuming that is the correct folder? I just wanted to make sure because the path didn't seem to be the same as the one you wrote above but I realize that paths will vary from OS to OS.

And then once I add those files in there I am done?

Thanks in advance.
Also one more....it would be great if any of you guys that have used this scraper and mod a lot more could tell me what the best combination of settings you have found. I am just looking to get the basic data on the flim, good art and the RT score. Thanks again.
Reply
#8
(2012-09-12, 02:52)adrock0211 Wrote: Forgive my ignorance but when you say to unzip the files and add them to the userdata/addons folder I can't seem to find that path. I am running XBMC on a Windows 7 64 Bit computer. When I go to C:\Program Files (x86)\XBMC\UserData I only see a folder called PictureIcon and 4 XML docs. I DO have C:\Program Files (x86)\XBMC\addons with a bunch of metadata.xxxxxx.xxxxx in there so I am assuming that is the correct folder? I just wanted to make sure because the path didn't seem to be the same as the one you wrote above but I realize that paths will vary from OS to OS.

And then once I add those files in there I am done?

Thanks in advance.
Also one more....it would be great if any of you guys that have used this scraper and mod a lot more could tell me what the best combination of settings you have found. I am just looking to get the basic data on the flim, good art and the RT score. Thanks again.

Windows Vista/7: Users\%userprofile%\AppData\Roaming\XBMC\userdata\addons

http://wiki.xbmc.org/index.php?title=Userdata

Reply
#9
Great. I replaced a bunch of files in that other directory...I hope I didn't screw something up. Thanks for the info.
Reply
#10
(2012-09-12, 02:52)adrock0211 Wrote: Also one more....it would be great if any of you guys that have used this scraper and mod a lot more could tell me what the best combination of settings you have found. I am just looking to get the basic data on the flim, good art and the RT score. Thanks again.

Currently I'm using title USA/International, original title Italian (I'm Italian Big Grin), plot from IMDb (if I remember correctly), all sources for trailers and art, RT score with google search, all critics in rating and audience + special info in votes (I'm using a personal modification of refocus BIG to show these info correctly).
Reply
#11
Hey Zeppy, maybe it's my skin (currently I am running Alaska Revisited) but I can't get the Rotten Tomatoes score to show up on my movies. Does this mod work especially better with a particular skin? I also don't have a lot of movies in my library yet so to test this without ripping tons of movies can I put a "fake" movie file in the default location and the trick the scraper into looking for that info just to test the mod out?

Thanks.
Reply
#12
(2012-09-17, 22:54)adrock0211 Wrote: Hey Zeppy, maybe it's my skin (currently I am running Alaska Revisited) but I can't get the Rotten Tomatoes score to show up on my movies. Does this mod work especially better with a particular skin?

Which RT scores are you referring to? The usual score (the same one you can choose in Universal Scraper, like tomatometer) should be visible as long as Alaska Revisited uses $INFO[ListItem.RatingAndVotes] or $INFO[ListItem.Rating] to show it. If you're trying to use the additional scores I encode in the votes field, then you need a custom skin that can handle that encoding. I'm currently working on a mod for refocus BIG that handles these scores and shows icons as well (see screenshot in first post).
Also, consider that Alaska Revisited is not 100% compatible with xbmc 11: if you're using xmbc 11 Alaska Revisited may be failing you in showing the correct score; if instead you're using xmbc 10 be aware that I've modified the Universal Scraper that comes with xmbc 11.

(2012-09-17, 22:54)adrock0211 Wrote: I also don't have a lot of movies in my library yet so to test this without ripping tons of movies can I put a "fake" movie file in the default location and the trick the scraper into looking for that info just to test the mod out?

That should be pretty easy. Create a new folder wherever you feel more confortable and add dummy files with the naming convention used for dvds/blu rays. For example, you can create an empty file and name it The Avengers.dvd.disc. Then go to xbmc, add this source (I suggest you add a new source, separate from your movie library, so you can play around with settings - in the end movies will show up in the same category anyway), set the scraper and let it scrape your new fake movies. Once it's done, go to movies and check out the new entries.
You can later re-scrape the same fake movies if you change something in the settings, or add new dummy files and tell xbmc to check for new media for that specific source.
When you're satisfied with your tests, you can copy your settings over to your original library (and re-scrape your movies, if necessary), manually delete the fake entries from xbmc, remove the source and delete the dummy files.

(2012-09-17, 22:54)adrock0211 Wrote: Thanks.

Thank you for your feedback =)
Reply
#13
(2012-09-18, 09:01)zeppy Wrote: Which RT scores are you referring to? The usual score (the same one you can choose in Universal Scraper, like tomatometer) should be visible as long as Alaska Revisited uses $INFO[ListItem.RatingAndVotes] or $INFO[ListItem.Rating] to show it.

Forgive my ignorance but how would I check this? And I am not trying to get any additional ratings to show up...just the regular old tomato score is all that I am looking for.

(2012-09-18, 09:01)zeppy Wrote: That should be pretty easy. Create a new folder wherever you feel more confortable and add dummy files with the naming convention used for dvds/blu rays. For example, you can create an empty file and name it The Avengers.dvd.disc.

Thanks, I will try this out when I get home tonight to make sure it's not just a lack of title in my catalogue. One question on this though...my current naming convention is like this: American Psycho (2000).mkv Will this naming convention not scrape effectively? I read somewhere on another forum that this is a good naming convention specifically for this reason but in your example above you seem to have another naming convention. I just want to make sure I use the one that is going to scrape the best. I "think" I am ok though because it IS finding the title just fine and getting all the other data from IMDB etc...it's just not showing the tomato score. And to answer your question I am running XBMC 11.0 Eden. Thanks.

Reply
#14
(2012-09-18, 15:55)adrock0211 Wrote: Forgive my ignorance but how would I check this? And I am not trying to get any additional ratings to show up...just the regular old tomato score is all that I am looking for.

There's no easy way but to check out every xml file of the skin, unfortunately. But you could try to approach the problem from another side: use the standard skin, or another that is xbmc 11 compatible and see if the ratings show.
I tried Alaska Revisited myself some time ago, but indeed it showed some problems to due its incompatibility with xbmc 11, so my gut is telling me that that may be the cause.
Another approach may be using the standard Universal Scraper and check if you can see scores scraped using it.

(2012-09-18, 15:55)adrock0211 Wrote: Thanks, I will try this out when I get home tonight to make sure it's not just a lack of title in my catalogue. One question on this though...my current naming convention is like this: American Psycho (2000).mkv Will this naming convention not scrape effectively? I read somewhere on another forum that this is a good naming convention specifically for this reason but in your example above you seem to have another naming convention. I just want to make sure I use the one that is going to scrape the best. I "think" I am ok though because it IS finding the title just fine and getting all the other data from IMDB etc...it's just not showing the tomato score. And to answer your question I am running XBMC 11.0 Eden. Thanks.

That should work but, to be honest, I've found 2 or 3 examples (can't remember which right now) that gave some problems when I also added the year in the file name. It should be something related to the initial imdb search, though, which is a part of the code I haven't touched.
Reply
#15
(2012-09-12, 03:02)iheartnerdz Wrote: Windows Vista/7: Users\%userprofile%\AppData\Roaming\XBMC\userdata\addons

http://wiki.xbmc.org/index.php?title=Userdata

I have Users\%userprofile%\AppData\Roaming\XBMC\addons
and
Users\%userprofile%\AppData\Roaming\XBMC\userdata\addon_data

which one is the correct path to add the folders?

Thanks
Reply

Logout Mark Read Team Forum Stats Members Help
[MOD] Universal Scraper+ v2.1.61