[RELEASE] - Rating Update

m4x1m · 2011-08-09 18:33

Hi all, this is my first script in python language I hope you like it.

What does
Basically updates the ratings and the number of users who have voted of all your movies/tv shows (no single episodes) from imdb and also updates the top250 list.

How it works
Use the IMDbPHP Class locally to retrieve only the actual ratings and top250 without additional data. The XBMC database is only updated if necessary. The searches are made through IMDb ID, if in your database doesn't exist IMDb ID the script will fail. For the TV Shows the first launch could ask you to choose the correct TV Show from a list then the IMDb ID will be written in the database (the edit is possible by activating the option in the add-on settings).

Now with a simple GUI to control the script by menu!

[Image: ratingUpdate.png]

To install php-cgi binary see README file or read below

Code:

Linux Ubuntu Users:

- Install php5-cgi "sudo apt-get install php5-cgi".

- For other distributions check the name of package (could be different) and install it.

- Check the correct path (/usr/bin/ by default) in the add-on settings.

Windows Users:

- Download php-cgi from http://windows.php.net/download/ choose the last "VC9 x86 Non Thread Safe" zip package.

- Unzip in C:\php5\ or wherever you like.

- Update the correct path in the add-on settings.

Mac Users:

- Sorry, I don't have a Mac. Search in Google on how to install PHP :-D

- Update the correct path in the add-on settings.

For Skinners:

- To run single Movie update: <onclick>RunScript(script.ratingupdate,Single=Movie)</onclick>

- To run single TV Show update: <onclick>RunScript(script.ratingupdate,Single=TVShow)</onclick>

Check what has been done
You can check what has been done looking the xbmc.log. Every notice of the script starts with [Rating Update].

Known Bugs
For the Windows users who have whitespaces in their path the script will not work. For a temporary workaround please refer here.

Pay Attention
For mySql users please pay attention, now you must specify in configuration settings, the number of database version!

Download
script.ratingupdate-1.2.9.zip

Code:

v1.2.9:

- Added Portuguese (Brazil) language. Thanks to Wanilton.

- Little cleanup code.

v1.2.8:

- Improved unicode characters views (I hope).

- Improved searching of TV Show by name (if no result was found, opens keyboard. ESC for skip).

- Print in the XBMC log also movies and tv shows that are already updated.

- Fixed some minor bugs.

v1.2.7:

- Fixed unicode error when scraping a TV Shows.

v1.2.6:

- Updated IMDbPHP Version 2.1.4 to new Version 2.1.6.

v1.2.5:

- Fixed issue for compatibility with Frodo. For mySql users please pay attention, now you must specify in configuration settings, the number of database version!

- Updated IMDbPHP Version 2.1.2 to new Version 2.1.4.

- Others minor bugs fixed.

v1.2.4:

- Fixed error in dbutils.py that prevented to connect to mysql database.

v1.2.3:

- Added positions with leading zeros and year of the movies to the exported file.

v1.2.2:

- Added export of missing movies from the top250 list. Choose the path where to export the txt file in add-on settings otherwise the script will create it in his profile folder.

v1.2.1:

- Added buttons images in media folder for those skins that don't use the default names for these images.

- Added status and latest update date of checks under the icon.

v1.2.0:

- Added German translation. Thanks to solidsatras.

- Fixed typo in english translation.

- Fixed typo in dbutils.py that stopped the script.

v1.1.9:

- Added the ability to scrape a single TV Show (The code for the skinners is in the README file).

- Changed database query method from httpapi to raw for prevent xbmc freezes and added dependency for script.module.myconnpy for mysql users (thanks to ezechiel1917 for the hint).

- Moved 'import json' to 'import simplejson as json' and added dependency for script.module.simplejson to make the script more compatible.

- Solved issue related for double quote in path for the Windows users. This is a temporary hack because if there are whitespaces in the path the script will not work.

- Updated IMDbPHP Version 2.1.0 to new Version 2.1.2.

- Added a simple GUI to control the script by menu (can be disabled).

- Reorganized the resource folder with a different layout.

- Added multilanguage strings (Now is complete). If someone want to add a new language it's welcome.

v1.1.8:

- Added missing quotes to the executable path (To fix issue in MACOS).

v1.1.7:

- Fixed issue related MacOS (I hope, I don't have a MAC to test).

- Added more debug info (visible when debugging is enabled in xbmc: Settings -> System -> Debugging -> Enable debug logging).

v1.1.6:

- Fixed profile folder that doesn't exist on the first launch of the script.

- Fixed error for mysql database that made the script fail.

v1.1.5:

- Added the resume ability when user interrupting the scraping (only for movies and tv shows should also works in background if xbmc crash,

  I don't think works when IMDb is unreachable, but not tested).

- Added notify for the actual name of the tv show when you need to choose the correct tv show from the list.

- Added multilanguage strings (Not complete).

- Added in TV Shows number of users who have voted even if the skin doesn't show it.

- Fixed unicode error in TV Shows while scraping.

v1.1.4:

- Fixed problem with damned Windows path (For real).

v1.1.3:

- Reduced the number of queries to the XBMC database.

- Fixed problem with damned Windows path (I hope).

- Fixed some others minor bugs.

v1.1.2:

- Fixed some unicode errors.

- Added more check.

v1.1.1:

- Fixed bug that had blocked access to the IMDb site while scraping a single movie.

- Fixed other some minor bug.

- Improved code and output log.

v1.1.0:

- Changed the method of scraping. IMDbPHP now is run in a local system to prevent the ban of the server IP address by the IMDb staff.

  See README file to how to install php-cgi binary.

- Added the ability to scrape ratings of TV Shows (no single episodes, to many differences with thetvdb.com).

  The first launch could ask you to choose the correct TV Show from a list then the IMDb ID will be written in the database.

- Changed order of questions: 1. Top250 - 2. Movies - 3. TV Shows

- Fix possible freeze when scraping a single movie. Show result using notifications instead of dialog.ok (I hope)

- Added setting to disable single procedure, confirmation and to hide the progress dialog.

- Added setting to delete IMDb id from TV Shows so you can rescrape and choose another title if the first one chosen was wrong.

v1.0.5:

- Fixed again bug that now was adding -1 to the real position taking from Top250.

v1.0.4:

- Changed IMDb Api, now using imdbphp class (although it'is a little slower but ...) to retrieve only the actual ratings and top250 without additional data.

  If anyone knows a good free hosting service, or wants to host the php files (warning: possible increase of traffic), please contact me on the XBMC forum, currently the php class reside on a web space that isn't mine and I can't use it for a long time.

- Improved get_html_source function for rating scraping (now don't stop the process if the api site is unreachable but it will try 5 time then will continue).

- Fixed double log (adding & updated).

- Added the ability to place a button on DialogVideoInfo.xml to retrieve the rating of a single movie.

  This is the Run for the skinners: <onclick>RunScript(script.ratingupdate,Single=True)</onclick>

v1.0.3:

- Fixed bug that was adding +1 to the real position taking from Top250.

- Remove fanart image (I don't like anymore).

- Summary result and log information improved.

v1.0.2:

- Fixed print log value for old position for Top250.

- Fixde crash that occurs when api site returns a Parse Error.

v1.0.1:

- Fixed crash that occurs when api site returns N/A.

- Change print log about old and new positions for a clearer understanding.

- Split on two lines print result to fit in gui box.

v1.0.0:

- First script release.

**ronie** · 2011-08-09 19:11

the resources folder in the zip is empty ;-)

m4x1m · 2011-08-09 19:43

ronie Wrote:the resources folder in the zip is empty ;-)

Thanks ronie, my mistake in zip command line, I've uploaded again! Wink

Big_Noid · 2011-08-09 21:59

I get an error with the script:

Code:

INFO: -->Python script returned the following error<--

21:42:55 T:2124   ERROR: Error Type: <type 'exceptions.ValueError'>

21:42:55 T:2124   ERROR: Error Contents: invalid literal for float(): N/A

21:42:55 T:2124   ERROR: Traceback (most recent call last):

                                              File "C:\Program Files (x86)\XBMC_SVN\portable_data\addons\script.ratingupdate\resources\rating_scraper​.py", line 224, in <module>

                                                RatingUpdate()

                                              File "C:\Program Files (x86)\XBMC_SVN\portable_data\addons\script.ratingupdate\resources\rating_scraper​.py", line 161, in __init__

                                                complete = self.scan()

                                              File "C:\Program Files (x86)\XBMC_SVN\portable_data\addons\script.ratingupdate\resources\rating_scraper​.py", line 199, in scan

                                                votes = group(float(decode[u'Votes']))

                                            ValueError: invalid literal for float(): N/A

21:42:55 T:2124    INFO: -->End of Python script error report<--

ezechiel1917 · 2011-08-09 22:26

great work m4x1m Cool

love the speed, logs and visual progress of updates.

1) Do you know how often is that IMDB API site updated. It seems to be lagging behind IMDB rating/votes (my guess that is few months). :/ This unfortunately degrades the point of frequent updates. Isn't there available more up-to-date alternative api site?

2) imdb top250 list gets updated fine, but the log reports wrong old position of movies

Code:

21:44:04 T:2785008496 M:1863168000  NOTICE: [Rating Update] - 'The Shawshank Redemption': old position was 2, new position is 1

21:44:05 T:2785008496 M:1863180288  NOTICE: [Rating Update] - 'The Godfather': old position was 3, new position is 2

etc

even before shawshank was 1st, the godfather ... 2nd.

3) addon crashes when IMDB API site returns N/A rating. For example: http://www.imdbapi.com/?i=&t=elephant+white

Code:

22:03:17 T:2860501872 M:1871056896    INFO: -->Python script returned the following error<--

22:03:17 T:2860501872 M:1871056896   ERROR: Error Type: <type 'exceptions.ValueError'>

22:03:17 T:2860501872 M:1871056896   ERROR: Error Contents: invalid literal for float(): N/A

22:03:17 T:2860501872 M:1871056896   ERROR: Traceback (most recent call last):

                                              File "/home/xbmc/.xbmc/addons/script.ratingupdate/resources/rating_scraper.py", line 224, in <module>

                                                RatingUpdate()

                                              File "/home/xbmc/.xbmc/addons/script.ratingupdate/resources/rating_scraper.py", line 161, in __init__

                                                complete = self.scan()

                                              File "/home/xbmc/.xbmc/addons/script.ratingupdate/resources/rating_scraper.py", line 199, in scan

                                                votes = group(float(decode[u'Votes']))

                                            ValueError: invalid literal for float(): N/A

22:03:17 T:2860501872 M:1871056896    INFO: -->End of Python script error report<--

4) any chance for next version to optionally run updates on background?

m4x1m · 2011-08-10 14:29

ezechiel1917 Wrote:great work m4x1m love the speed, logs and visual progress of updates.

Thanks!

ezechiel1917 Wrote:1) Do you know how often is that IMDB API site updated. It seems to be lagging behind IMDB rating/votes (my guess that is few months). :/ This unfortunately degrades the point of frequent updates. Isn't there available more up-to-date alternative api site?

This service is updated every day from the IMDB data files -> http://imdbapi.poromenos.org/

Unfortunately I found only this two api site that works good, IMDb has a restrictive policy for the scraping.

ezechiel1917 Wrote:2) imdb top250 list gets updated fine, but the log reports wrong old position of movies

Code:

21:44:04 T:2785008496 M:1863168000 NOTICE: [Rating Update] - 'The Shawshank Redemption': old position was 2, new position is 1 21:44:05 T:2785008496 M:1863180288 NOTICE: [Rating Update] - 'The Godfather': old position was 3, new position is 2

etc

even before shawshank was 1st, the godfather ... 2nd.

The old position is read from xbmc database, i can't get the old position from the api site. I will change the print log in 'Godfather': old position in your db was 3, new position from api site is 2'

ezechiel1917 Wrote:3) addon crashes when IMDB API site returns N/A rating. For example: http://www.imdbapi.com/?i=&t=elephant+white

Code:

22:03:17 T:2860501872 M:1871056896 INFO: -->Python script returned the following error<-- 22:03:17 T:2860501872 M:1871056896 ERROR: Error Type: <type 'exceptions.ValueError'> 22:03:17 T:2860501872 M:1871056896 ERROR: Error Contents: invalid literal for float(): N/A 22:03:17 T:2860501872 M:1871056896 ERROR: Traceback (most recent call last): File "/home/xbmc/.xbmc/addons/script.ratingupdate/resources/rating_scraper.py", line 224, in <module> RatingUpdate() File "/home/xbmc/.xbmc/addons/script.ratingupdate/resources/rating_scraper.py", line 161, in __init__ complete = self.scan() File "/home/xbmc/.xbmc/addons/script.ratingupdate/resources/rating_scraper.py", line 199, in scan votes = group(float(decode[u'Votes'])) ValueError: invalid literal for float(): N/A 22:03:17 T:2860501872 M:1871056896 INFO: -->End of Python script error report<--

@Big_Noid

I will fix it soon.

ezechiel1917 Wrote:4) any chance for next version to optionally run updates on background?

I'll see what I can do, I don't have full knowledge of Python will be studying it.

m4x1m · 2011-08-10 15:16

Code:

v 1.0.1:

- Fix crash that occurs when api site returns N/A

- Change print log about old and new positions for a clearer understanding

- Split on two lines print result to fit in gui box

Download:
script.ratingupdate-1.0.1.zip

Please test it, I don't have movies without ratings Sad

ezechiel1917 · 2011-08-10 15:54

m4x1m Wrote:This service is updated every day from the IMDB data files -> http://imdbapi.poromenos.org/

Unfortunately I found only this two api site that works good, IMDb has a restrictive policy for the scraping.

Unfortunately that poromenos.org api site only returns results for tv shows. Too bad there isn't reliable alternative for movies Sad

I don't need to have daily updates but from query results of current API site, their data are way too old for my taste. E.g. that Elephant White has N/A with IMDB API but IMDB.com has already over 3000 votes. Sad

m4x1m Wrote:The old position is read from xbmc database, i can't get the old position from the api site. I will change the print log in 'Godfather': old position in your db was 3, new position from api site is 2'

Actually it was 2nd in my XBMC db, that's why I was reporting it Wink

m4x1m · 2011-08-10 16:42

ezechiel1917 Wrote:Unfortunately that poromenos.org api site only returns results for tv shows. Too bad there isn't reliable alternative for movies I don't need to have daily updates but from query results of current API site, their data are way too old for my taste. E.g. that Elephant White has N/A with IMDB API but IMDB.com has already over 3000 votes.

http://www.deanclatworthy.com/imdb/

this api site seems to be very updated, but the query is only by the title and not by id. I will try to replace this with that other.

ezechiel1917 Wrote:Actually it was 2nd in my XBMC db, that's why I was reporting it

Yes, you're right, I fix it in the next release!

**gabbott** · 2011-08-10 17:07

Thanks for this. Giving it a try right now, have about 720 movies for it to go through. Will report back how it goes.

ronie Team-XBMC Member Posts: 8,247 Joined: Jan 2009 Reputation: 108	2011-08-09 19:11 Post: #2 the resources folder in the zip is empty ;-) Always read the XBMC online-manual, FAQ and search the forum before posting. Do not PM or e-mail XBMC-Team members directly asking for support. Read/follow the forum rules. For troubleshooting and bug reporting please make sure you read this first.
	find quote

m4x1m Senior Member Posts: 202 Joined: Mar 2011 Reputation: 12	2011-08-09 19:43 Post: #3 ronie Wrote:the resources folder in the zip is empty ;-) Thanks ronie, my mistake in zip command line, I've uploaded again! LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB WI-FI Running Ubuntu Precise 12.04.01 - FRODO 12.00 - Transparency! Movies: 892 - TV Show: 97 - Albums: 1143
	find quote

gabbott Team-XBMC Member Posts: 1,547 Joined: Jul 2007 Reputation: 26	2011-08-10 17:07 Post: #10 Thanks for this. Giving it a try right now, have about 720 movies for it to go through. Will report back how it goes.
	find quote