[RELEASE] - Thumbnails Cleaner v1.1.3

  Thread Rating:
  • 3 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
SpaceMonkey Offline
Posting Freak
Posts: 1,162
Joined: Aug 2008
Reputation: 24
Location: Finland
Post: #16
It is <name>xbmc_video</name> + the actual db version read from the version table. So with this config it's xbmc_video75. I'm fairly certain most of the mysql db users follow this same naming scheme and the advanced settings.xml looks the same.

If <name>blaablaa</name> is not used it's MyVideos75, even in mysql.
(This post was last modified: 2013-03-14 21:55 by SpaceMonkey.)
find quote
m4x1m Offline
Fan
Posts: 311
Joined: Mar 2011
Reputation: 23
Location: Italy
Post: #17
(2013-03-14 21:54)toiva Wrote:  It is <name>xbmc_video</name> + the actual db version read from the version table. So with this config it's xbmc_video75. I'm fairly certain most of the mysql db users follow this same naming scheme and the advanced settings.xml looks the same.

If <name>blaablaa</name> is not used it's MyVideos75, even in mysql.

Ok dbutils.py modified for reflect this situation, please replace with this one

LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB
Running Ubuntu Precise 12.04.04 - FRODO 12.03 - Transparency!
Movies: 996 - TV Show: 98 - Albums: 1161

My Script: Rating Update - Thumbnails Cleaner
find quote
SpaceMonkey Offline
Posting Freak
Posts: 1,162
Joined: Aug 2008
Reputation: 24
Location: Finland
Post: #18
Now it gets stuck at finalizing phase when running the simulation.

Code:
22:59:09 T:4632645632   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.OSError'>
                                            Error Contents: (1, 'Operation not permitted', '/Users/anssi/Pictures/Backdrops')
                                            Traceback (most recent call last):
                                              File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/gui.py", line 155, in onClick
                                                elif controlId == self.idSimulate: Cleaner( self ).doClean(True)
                                              File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/cleaner.py", line 301, in doClean
                                                self.finalizeClean( doSimulate )
                                              File "/Users/anssi/Library/Application Support/XBMC/addons/script.thumbnailscleaner/resources/lib/cleaner.py", line 334, in finalizeClean
                                                for f in getFiles: os.remove(f)
                                            OSError: (1, 'Operation not permitted', '/Users/anssi/Pictures/Backdrops')
                                            -->End of Python script error report<--

I've set /Users/anssi/Pictures/ as a target, is it trying to recursively delete the child directories? I also tried /Users/anssi/Pictures/test but it still fails with the same error.

Second issue, it does not seem to detect my music video posters. If you use Concertdb scraper it scrapes standard posters from TMDB. So music videos can use either poster or a thumb, depending on a scraper used.

Thanks for the efforts so far, we shall continue tomorrow.
find quote
m4x1m Offline
Fan
Posts: 311
Joined: Mar 2011
Reputation: 23
Location: Italy
Post: #19
(2013-03-14 22:56)toiva Wrote:  I've set /Users/anssi/Pictures/ as a target, is it trying to recursively delete the child directories? I also tried /Users/anssi/Pictures/test but it still fails with the same error.

Second issue, it does not seem to detect my music video posters. If you use Concertdb scraper it scrapes standard posters from TMDB. So music videos can use either poster or a thumb, depending on a scraper used.

Thanks for the efforts so far, we shall continue tomorrow.

I changed the method to move files. Now will be deleted only the files that already exist in the destination folder, instead to entirely empty the folder. More safely. (Python: shutil.copy2 overwrites the files, shutil.move no Confused )

Thanks for your time!

Download
script.thumbnailscleaner-1.0.4.zip

Code:
v1.0.4:
- Fixed all problems related to mysql database (for real).
- Added check for music video posters.
- Fixed issue when moving files in destination folder that already exist.

LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB
Running Ubuntu Precise 12.04.04 - FRODO 12.03 - Transparency!
Movies: 996 - TV Show: 98 - Albums: 1161

My Script: Rating Update - Thumbnails Cleaner
(This post was last modified: 2013-03-15 21:01 by m4x1m.)
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #20
Nice work. May I suggest some changes:

1. Don't bother looking in MyVideos75.db. There's no need to do so. If you MUST look in there, then just use JSON-RPC to grab what you need - that way you're immune to db layout changes.

2. Do bother looking in Textures13.db. It's a local sqlite database only. The layout can and will change, but you can't do anything about that for now, so don't bother Wink All you need do is compare what's there with what's on disk.

3. Further, you can then start doing cleanup based on the time and usage information stored in that database. e.g. you could clean up stuff that hasn't been used for ages. Note that this gets around having to play in the videodb or musicdb's - stale entries in Textures13.db gets cleaned out automatically with time.

4. Note that XBMC operates under the assumption that Textures.db is king - i.e. it doesn't matter if it's on disk, if it's not in the database it'll be recached. Thus, to recache stuff, just remove the entry in Textures.db (assuming ofc that the original URL is still valid). Further, this implies that anything on disk and not in Textures.db can be removed, and anything not on disk but in Textures.db is a stale entry in Textures.db so should be removed.

Lastly, once you've come up with a good system, we'll look to expose whatever you need via JSON-RPC so you're then immune to db-layout issues and no longer need a database connection. I suspect all you really need is a way to grab everything from the texture (joined size) tables?

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
(This post was last modified: 2013-03-15 03:09 by jmarshall.)
find quote
m4x1m Offline
Fan
Posts: 311
Joined: Mar 2011
Reputation: 23
Location: Italy
Post: #21
(2013-03-15 03:06)jmarshall Wrote:  1. Don't bother looking in MyVideos75.db. There's no need to do so. If you MUST look in there, then just use JSON-RPC to grab what you need - that way you're immune to db layout changes.

From MyVideos75.db (MyMusic32) i get the the info from the art table, the script generates the hash and then excludes the images from the list of files to be deleted. Yes, I can convert the query to JSON, so I'll study to do this. (It's possible to get the content of the art table with JSON? I couldn't even get the version number in my previous attempt.)

(2013-03-15 03:06)jmarshall Wrote:  2. Do bother looking in Textures13.db. It's a local sqlite database only. The layout can and will change, but you can't do anything about that for now, so don't bother Wink *All you need do is compare what's there with what's on disk.*

Yes, in fact I do this, exactly as MyVideos and MyMusic.

(2013-03-15 03:06)jmarshall Wrote:  3. Further, you can then start doing cleanup based on the time and usage information stored in that database. e.g. you could clean up stuff that hasn't been used for ages. Note that this gets around having to play in the videodb or musicdb's - stale entries in Textures13.db gets cleaned out automatically with time.

In the settings I added a slider to set the days. I compare the days with the lasthashcheck field, then deletes the old. I added an option to delete the fields that have an empty lasthashcheck (I noticed that normally have an http:// url)

(2013-03-15 03:06)jmarshall Wrote:  4. Note that XBMC operates under the assumption that Textures.db is king - i.e. it doesn't matter if it's on disk, if it's not in the database it'll be recached. Thus, to recache stuff, just remove the entry in Textures.db (assuming ofc that the original URL is still valid). Further, this implies that anything on disk and not in Textures.db can be removed, and anything not on disk but in Textures.db is a stale entry in Textures.db so should be removed.

Yes, I know this.

(2013-03-15 03:06)jmarshall Wrote:  Lastly, once you've come up with a good system, we'll look to expose whatever you need via JSON-RPC so you're then immune to db-layout issues and no longer need a database connection. I suspect all you really need is a way to grab everything from the texture (joined size) tables?

I'll work to do this!

In conclusion. I read all files present in the Thumbnails folder. I get arts info from the MyVideos db, calculates hash and excludes the files. Same thing with the MyMusic db. Process the textures table, deletes all fields with empty lasthashcheck (option), deletes all fields that are older than X days (optional), then compares the fileds that are remained with my list of files excluding the files that are present, and lastly copy/move the remained files, that can be safely to deleted. I think this is the correct way, but I could be wrong.

Thanks!

UPDATE: OK, I get the arts with JSON, except for the actors.
1. How to I get the actors thumbs?
2. The season-all-poster is missing when query GetSeasons.
2. Can I call the textures13.db with JSON?

LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB
Running Ubuntu Precise 12.04.04 - FRODO 12.03 - Transparency!
Movies: 996 - TV Show: 98 - Albums: 1161

My Script: Rating Update - Thumbnails Cleaner
(This post was last modified: 2013-03-15 20:05 by m4x1m.)
find quote
Milhouse Offline
Team-Kodi Member
Posts: 4,208
Joined: Jan 2011
Reputation: 95
Post: #22
(2013-03-15 12:31)m4x1m Wrote:  UPDATE: OK, I get the arts with JSON, except for the actors.
1. How to I get the actors thumbs?
2. The season-all-poster is missing when query GetSeasons.
3. Can I call the textures13.db with JSON?

1) By adding the "cast" property to the VideoLibrary.GetMovies, VideoLibrary.GetTVShows and VideoLibrary.GetExpisodes methods. However JSON is a bit flaky with respect to cast thumbs as it will only return cast thumbs for manually scraped movies/tvshows, and won't return cast thumbs that are cached dynamically. There's a disconnect somewhere, probably highlighted by enabling "Download actor thumbnails" after performing a full scrape (you probably wouldn't notice the problem while scraping with it enabled).

2) Yes, probably because there is no concept of a "season all" in the database, only Season #0 (Special) through to Season #n. Unless there is a change forthcoming, you'll probably need to hack in season-all support. I suppose JSON could implement a "season-all" season somehow, perhaps using seasonid -1 (ugh), though this seasonid when calling GetEpisodes would not be valid so you'd need to special case it.

3) No, you need to use the sqlite3 module.

Texture Cache Maintenance Utility: Preload your texture cache for optimal UI performance. Remotely manage media libraries. Purge unused artwork to free up space. Find missing media. Configurable QA check to highlight metadata issues. Aid in diagnosis of library and cache related problems.
find quote
m4x1m Offline
Fan
Posts: 311
Joined: Mar 2011
Reputation: 23
Location: Italy
Post: #23
(2013-03-18 10:39)MilhouseVH Wrote:  
(2013-03-15 12:31)m4x1m Wrote:  UPDATE: OK, I get the arts with JSON, except for the actors.
1. How to I get the actors thumbs?
2. The season-all-poster is missing when query GetSeasons.
3. Can I call the textures13.db with JSON?

1) By adding the "cast" property to the VideoLibrary.GetMovies, VideoLibrary.GetTVShows and VideoLibrary.GetExpisodes methods. However JSON is a bit flaky with respect to cast thumbs as it will only return cast thumbs for manually scraped movies/tvshows, and won't return cast thumbs that are cached dynamically. There's a disconnect somewhere, probably highlighted by enabling "Download actor thumbnails" after performing a full scrape (you probably wouldn't notice the problem while scraping with it enabled).

2) Yes, probably because there is no concept of a "season all" in the database, only Season #0 (Special) through to Season #n. Unless there is a change forthcoming, you'll probably need to hack in season-all support. I suppose JSON could implement a "season-all" season somehow, perhaps using seasonid -1 (ugh), though this seasonid when calling GetEpisodes would not be valid so you'd need to special case it.

3) No, you need to use the sqlite3 module.

Thanks for the reply!

I spent long time testing json, but definitely, for me, in this case is better to use a raw approach to the database. To get season arts or episode thumbs is necessary, with json, create a double loop that slows the operation. With a raw approach I can get all info that I needed, directly, also the actors thumbs. Never mind if I have to keep up with the changes in the database layout.

LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB
Running Ubuntu Precise 12.04.04 - FRODO 12.03 - Transparency!
Movies: 996 - TV Show: 98 - Albums: 1161

My Script: Rating Update - Thumbnails Cleaner
(This post was last modified: 2013-03-18 11:58 by m4x1m.)
find quote
patria o muerte Offline
Member
Posts: 81
Joined: Feb 2013
Reputation: 0
Post: #24
What a addon!
I saved about 119 mb of disk space.
Very useful.
Thanks!
find quote
DaDuck Offline
Junior Member
Posts: 7
Joined: Feb 2013
Reputation: 0
Post: #25
Is there any way to set the cleaner to remove all the episodes thumbs?
find quote
m4x1m Offline
Fan
Posts: 311
Joined: Mar 2011
Reputation: 23
Location: Italy
Post: #26
(2013-03-18 12:14)patria o muerte Wrote:  What a addon!
I saved about 119 mb of disk space.
Very useful.
Thanks!

Big Grin

(2013-03-18 13:01)DaDuck Wrote:  Is there any way to set the cleaner to remove all the episodes thumbs?

At the moment no, but in the future I would to add more options, but first I want to pick up some tips and be sure that everything works well!

LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB
Running Ubuntu Precise 12.04.04 - FRODO 12.03 - Transparency!
Movies: 996 - TV Show: 98 - Albums: 1161

My Script: Rating Update - Thumbnails Cleaner
find quote
DaDuck Offline
Junior Member
Posts: 7
Joined: Feb 2013
Reputation: 0
Post: #27
Okay

I have found a bug (?) anyway if you have this line in advanced settings the script don't work

<hideemptyseries>true</hideemptyseries>
(This post was last modified: 2013-03-18 15:39 by DaDuck.)
find quote
m4x1m Offline
Fan
Posts: 311
Joined: Mar 2011
Reputation: 23
Location: Italy
Post: #28
(2013-03-18 15:38)DaDuck Wrote:  Okay

I have found a bug (?) anyway if you have this line in advanced settings the script don't work

<hideemptyseries>true</hideemptyseries>

If in advancedsettings.xml there is an extraneous char like # (used normally as comment) the script fails because this char isn't part of xml format, for comment in xml uses <!-- -->. I think your problem is this, but if you want, you can send me your advancedsettings.xml in private or post the relevant code here. In reference to the error that you posted previously.

LG M227WD - ASUS S1-AT5NM10E - LOGITECH HARMONY 300i - MINI KEYBOARD USB
Running Ubuntu Precise 12.04.04 - FRODO 12.03 - Transparency!
Movies: 996 - TV Show: 98 - Albums: 1161

My Script: Rating Update - Thumbnails Cleaner
find quote
SpaceMonkey Offline
Posting Freak
Posts: 1,162
Joined: Aug 2008
Reputation: 24
Location: Finland
Post: #29
I use the same advanced setting and did not have any problem, atleast i didn't notice any. And just to confirm, it seems to work fine on mysql table now.
(This post was last modified: 2013-03-18 17:20 by SpaceMonkey.)
find quote
DaDuck Offline
Junior Member
Posts: 7
Joined: Feb 2013
Reputation: 0
Post: #30
Quote:If in advancedsettings.xml there is an extraneous char like # (used normally as comment) the script fails because this char isn't part of xml format, for comment in xml uses <!-- -->. I think your problem is this, but if you want, you can send me your advancedsettings.xml in private or post the relevant code here. In reference to the error that you posted previously.


I cleaned up me advanced settings works like a charm now
(This post was last modified: 2013-03-18 18:35 by DaDuck.)
find quote
Post Reply