AppleTV 2 (black) - Purging Thumbnails, Stale Thumbnails

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
ctawn Offline
Senior Member
Posts: 260
Joined: May 2009
Reputation: 0
Post: #1
I just moved over most of my Userdata folder from a Mac to an ATV2 without noticeable issue so far (which is fabulous), except for one thing: the size of the Thumbnail folder (>1GB). Storage is a concern on the ATV2 in a way it wasn't on the Mac.

Does anyone have any tips on how to reduce the size of this folder?

How and when does XBMC purge out-of-date thumbnails? (I know there are tons of completely random thumbnails in there.)

When you select fanart and thumbails for the library, using images that reside on a network share, do the original size images get cached locally as thumbnails too?

How about add-ons? (I discovered just randomly browsing some composite images from an old iPhoto library for example, clearly no longer needed).

Thanks for any insight (or links to relevant other threads).
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 25,700
Joined: Oct 2003
Reputation: 169
Post: #2
http://lmgtfy.com/?q=Stale+Thumbnails+site%3Axbmc.org

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
ctawn Offline
Senior Member
Posts: 260
Joined: May 2009
Reputation: 0
Post: #3
lol, cheeky monkey. obviously I can google too. I'll look at those results, but it's not immediately obvious what one should search for in this case. "Stale"? I mean who uses that term. Smile

OK, the first result, other than this thread, is 23 pages. I've spent 2-3 days migrating over from Mac to ATV2 (including jailbreaking and other irritating issues), and just spent 2 hours posting some summary findings in these forums for the benefit of others.* So some summary answers would be highly appreciated. (Realizing that jmarshall may be too busy to do that. But maybe some other kind soul.)

Note also that I did look into the mySQL database sharing option, but for the time being have ruled it out for complexity and speed concerns.

* noted summary posts today by me:
http://forum.xbmc.org/showthread.php?tid=95093
http://forum.xbmc.org/showthread.php?tid...pid1082288
(This post was last modified: 2012-04-20 23:31 by ctawn.)
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 25,700
Joined: Oct 2003
Reputation: 169
Post: #4
You used Stale Smile I didn't think it would hit anything but was pleasantly surprised (moral of the story: let google do the work!)

Summary:

1. There's no simple way.
2. For anything other than music and videos, you should be able to delete directly as long as you also kill Textures.db. The thumbs will then re-generate as needed. Alternatively you could do something smarter by querying the Textures.db file (version 6 is eden IIRC) and eliminating all those not used for a while.
3. For music and video thumbs there's no simple way, but there's a script in the thread you reference (2nd post) that is supposed to do something about it - no idea if it works or not Smile
4. Number 3 will be solved for video during the May merge window (will drop videos into case 2). Music will come when I get some time to do it.
5. Lastly, number 2 will be solved by XBMC automatically cleaning things out as needed. Again, when I get some time to do it.

If anyone wants to work on number 5 (eg come up with some reasonable queries based on what's in the db already) then feel free, it's quite possible that a script could do this until I get around to it.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
ctawn Offline
Senior Member
Posts: 260
Joined: May 2009
Reputation: 0
Post: #5
Thanks for your response here. I've been trying to understand it but so far am afraid I'm a little in the dark still. I tried downloading the Eden version of the script you mention in #3, but haven't got very far, as it's set up mostly for Windows. (I have a Mac). Rather stuck on how to configure JSON and how to execute it. I've asked for help in that thread on that.

Regarding #2, so you're saying I can just delete any files or folders found in /userdata/Thumbails except /Video and /Music. But if I delete even one file there I have to delete the entire Textures6.db also? (Unless I get "smart" Smile What if I'm not smart and just delete image files without touching the Textures6.db?

Given #2 & #3, I am inferring that the index of images for videos and music is not kept in Textures6.db, but rather in MyMusic18.db and MyVideos60.db and that the script might handle those. Correct?

Given #4, what will it mean if I wait until this "Merge" in May? What will that let me do? I assume it will also be valid for the ATV2?

As an aside, I take it now that the earlier number sequences for the MyMusic and myVideos .db files were hold overs from earlier versions of XBMC and can safely be archived or deleted?

Finally, I would be happy to purge the Music component of my db entirely; it was partially indexed once but I don't play music with XBMC. In that case would I just delete the MyMusic18.db and also the /Thumbnails/Music folder? (Saving only like 40 MB).

Thanks!
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 25,700
Joined: Oct 2003
Reputation: 169
Post: #6
Regarding 2, if you delete one file then XBMC will think the file is there and you'll be left with a blank icon.

The index is not kept in MyMusic18.db (well, actually, they are, just what the cached version is, which can also be determined from the album/artist/song path etc.)

Video thumbs aren't kept anywhere - it all depends on the path of the movie/show/season etc.

Assuming you get a nightly build after my changes go in (in May) then your video thumbs will be cached (re-cached in fact) and fall into case 2.

Yes on the music.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
mselley Offline
Junior Member
Posts: 28
Joined: Oct 2011
Reputation: 0
Post: #7
Hi Jonathan,

Just on the video thumbs caching,
do you have any ideas when in may it will be available in the nightly builds?
Also is it possible to move these to a mysql database store?
Could you help direct me to the sorce code for handling this im curious to have a browse.

Great Work!

Thanks,
Mark
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 25,700
Joined: Oct 2003
Reputation: 169
Post: #8
The pull req has not yet been made. I'm hoping to do up one for the needed changes before this hits today/tonight, and then another for the videolib code later.

You can see the code in my github branches - see the video_thumbs_to_cache branch:

https://github.com/jmarshallnz/xbmc/branches

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Dito Offline
Junior Member
Posts: 8
Joined: Feb 2012
Reputation: 0
Post: #9
This sounds like a reasonably good idea for a Program add-on: Artwork Cleaner-Upper.

I don't know a whole lot about add-on development yet, but I'm a reasonably experienced programmer.

I suppose the way to do this would be to collect all the image filenames stored in User/Library/Preferences/XBMC/thumbails/(Video?) and compare against the assigned image(s) for items in the video database.

I don't know how memory-intensive this would be, which could be a problem on the reasonably low powered ATV2. But I think it could (theoretically) work. Might have to go one directory at a time (A, B, ... 8, 9) to avoid having too large of a list to compare against.

Just out of curiosity, why DO you guys save image filenames as those awful hex strings? I'm sure it's got something to do with efficiency and/or database storage space, but I secretly think you do it to make our lives difficult. Smile
find quote
Martijn Offline
Team-XBMC
Posts: 10,307
Joined: Jul 2011
Reputation: 154
Location: Dawn of time
Post: #10
I was thinking of making such add-on but never got around it and was hoping this would be done by XBMC itself in future Tongue

Always read the XBMC online-manual, FAQ and search the forums before posting.
Do NOT e-mail Team-XBMC members asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting, make sure you read this first

For your mediacenter artwork go to
[Image: fanarttv.png]
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 25,700
Joined: Oct 2003
Reputation: 169
Post: #11
Read my posts above, particularly point 2 and 5, then take a nosy in Textures6.db.

It's all reasonably self-explanatory, and would be rather trivial to do a proper cleanup function once everything is using that.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
Dito Offline
Junior Member
Posts: 8
Joined: Feb 2012
Reputation: 0
Post: #12
Clever - I hadn't realized the textures were also referenced in a database. That's okay; I like databases. :-)

While I'm sure I could find the answer in the source code, I hope you'll forgive me for just asking: when does the "usecount" field get updated? I've got one entry that's been "used" 24 times, but it's the icon for some Chinese add-on from the official repository.

Is it the number of places a particular image appears somewhere in the system? Is it the number of times that image has been displayed on-screen? If it was the latter, I'd assume the usecount would be much higher.

This could be more complicated than I thought... but it's easy enough to backup some .db files along with my Thumbnails folder. Fire up the Python editor and do a little tinkering... heh.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 25,700
Joined: Oct 2003
Reputation: 169
Post: #13
It's the usecount, but in Eden it gets reset whenever we check the hash (24 hour minimum, as you view).

This has changed in master, so usecount is actual usecount of the image since it was cached.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
ctawn Offline
Senior Member
Posts: 260
Joined: May 2009
Reputation: 0
Post: #14
Revisiting this thread, any progress on a "cleaner-upper"?

I notice that in upgrading from Eden to Frodo, the entire artwork cache is rebuilt. Are the old thumbnails not removed? The size of my thumbnails folder has increased nearly 30%! That's a lot of waste.
find quote