FRODO Thumbnail sync storage
#46
Well... running the media manager pushes images to the same location as the media that is a given. This is fine if you have 1 or 2 disks but what if you have 50 or 100. Every time you run the media manager to refresh art it will cause the same spin up. After that you can (edge case but it exists) have new art called the same name in the same location with the same time stamp so there is absolutely no way XBMC can know it has changed.

On top of that it completely breaks the intuitiveness of refreshing images via XBMC.

I dont disagree that it has a nice point and click appeal for windows users but I see no advantage to using media managers when all they do is what XBMC can do already.
Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#47
XBMC before v12 never refreshed images unless you did it manually through the gui one-by-one for every movie. XBMC never knows when the image has changed. Media managers don't refresh every single image unless you specifically tell it to do so. Using images along side of the media doesn't change anything regarding refreshing images in XBMC.

The reason this was suggested to you was because it was one way to only download once over the internet, and then have the rest of the copying happen locally on the network.
Reply
#48
I understand the reason for the suggestion but I cant ask OE users to go away buy and install a copy of windows when you can do almost the same thing natively using rsync. On going is different as obviously this offers a path to replace the heavily used, but now deprecated, central art share whilst reducing the burden on duplicate file downloading at the cost of more spin ups.

However I am 100% certain that people were still going to do path sub anyway but now that I know definitely that this will not stop the duplicate file downloading I have real reasons to tell the many users they need to redo their setup and why. Prior to this the user feedback I was getting was that path sub was still working for them so they were still doing it regardless of what the XBMC wiki says.

ta
Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#49
XBMC can export the images itself for free... Set up one XBMC box, tell it to export the library as multiple files, and there you go. The "cost of more spin ups" is again, a one time deal.

I guess what I'm saying is, I can't see this being defined as inefficient when compared to path substitution.

Like I said before, they can just copy the textures db file over, because that's what's keeping track of what's been downloaded or not for images. If each XBMC client has the same thumbnails, the same textures db, and the same video db (via MySQL or otherwise) then it will not attempt to download the images again.
Reply
#50
(2012-11-21, 11:43)Ned Scott Wrote: XBMC can export the images itself for free... Set up one XBMC box, tell it to export the library as multiple files, and there you go. The "cost of more spin ups" is again, a one time deal.
...

Yeah its only a one time deal if you do it one time. But every time you add more media you would need to do it all again... hence my point

The issue isnt really the initial migration from Eden to Frodo that can be trivially covered either using "this one time deal" or using rsync etc ... it is the current incorrect perception that central shared art via path sub still works in Frodo even though it is not recommended. Yes it still works but as ascertained earlier even though you have an image unless textures.db knows you have it XBMC will grab it again. This is the bit that people currently generally don't know.

Add into the mix that some arches scale down images and it explains why people are companing that art on device X is all of a sudden low resolution. It looks like HD fanart is being replaced by lower res art when textures.db catches up on lower spec devices.

I think I have all the information i need. People that still have to do path sub (i.e. the large number of OE users that have art caches larger than their HDD capacity) will just have to have one path sub cache per device and libe with duplicates on their central store.

Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#51
(2012-11-14, 00:09)jmarshall Wrote: 1. Update a client. You'll be prompted to rescan video art when you go into videos.
2a. Optionally, copy userdata/thumbnails (you can drop the video and music subdirs) from the server to all clients.
2b. Or remove textures.db on that client.
3. Once done, remove path substitution.
4. Update all other clients.

If you really want to keep art on the NAS (not recommended) then just do what you've always done. Note that all clients will "recache" textures while they update their textures.db, so you could potentially end up with corrupt art files, though this is relatively unlikely.

Cheers,
Jonathan

My setup includes:
One Windows computer with Eden (call this 1st client)
Three xboxes running Eden (call this 2nd client and so on)
Using shared mySQL, and locally stored artwork on each client and I keep artwork exported separately updated

To make sure I understand this, for my setup, then:

1. Shut down all clients except 1st client
2. Upgrade 1st client to Frodo from Eden, no need to change my advandesettings since I don't use pathsubstitution
3. Start Frodo, and it will recreate new mySQL database, and the new texture.db will be created using my artwork that's stored locally and not the URL for the artwork? I guess that this is incorrect, it will use the URL no matter what?
4. 1st client should be up and ready to go.
5. Then on my 2nd client and so on, remove advancedsettings update to Frodo
5. Copy the first 1st client's advancedsettings.xml, artwork folder and texture.db into the 2nd client
6. Restart 2nd client and it should be fully up and running, with no need to create or download any artwork because of step 5
7. Any time I add a new movie to 1st client, 2nd client (and so on) will use the URL to get the same artwork (unless I have already done the export separately)?

If this is all correct, then it will make things simpler than what I do now, although what I do now is pretty simple.
Reply
#52
(2012-11-21, 10:39)xexe Wrote: j114 thanks for taking the time to reply. Unfortunately I cannot recommend the use of external media and nfo tools as OpenELEC is intended to be a native XBMC tool. Also this approach is definitely not best practice since it has other implications. For instance for me if I was to follow this approach instead of one cache disk spinning up it is theoretically possible that XBMC would cause the spin up of over 60 hard disks all in the name of retrieving a bunch of tiny image files. Also not all file systems are configured to save the modification time of an image which means for some to know if an image is new the image itself would need to be read (as the file name and date stamp may not have changed). I am sure this approach works for many people but its not for us. ta Smile

xexe, not to be argumentative or start a debate, the assumption that everyone using OE don't have a Windows machine is as inaccurate as the assumption that they all do (I have XBMC on multiple platforms, including OE). This really isn't about platforms at any rate. There are (fewer) options for other platforms as well, the 3 I referenced were just the ones on the top of my head at the time as they are 3 of the more popular choices. Unfortunately, like most things *nix, the options and features are slimmer.

As far as the point about disk spin ups, I should point out that my "NAS" I am referencing is actually an 18 disk unRAID array hosted on my ESXi box which also hosts the Windows VM's that contains my "master" copies of XBMC and media scraper. I have my scraper configured to monitor my media collection and automatically process any new content. XBMC then scans for new content at regular intervals and adds my already processed content (so no Internet scraping of it's own).

For the thumbnails, If you update art on one client, then simply sync all of your cache directories to a single master cache directory. Maybe I am being thick this morning and just not seeing the cause for any excessive increase in drive spin ups using multiple cache directories on a single drive. I'm certainly not seeing it in my environment, but that's not to say that everyone has things configured the way I do either.

I call it a best practice to store all metadata and artwork alongside the actual media because it makes it far easier to migrate across platforms and frontends, as well as saving on Internet bandwidth consumption now and in the future. Obviously, the goal is to find a solution that works for you with your unique environment and parameters, so my "best practice" might not be yours.

Ned did a superb job of pointing out the rest, so I'll not beat a dead horse.
Reply
#53
(2012-11-21, 19:30)j114 Wrote: xexe, not to be argumentative or start a debate, the assumption that everyone using OE don't have a Windows machine is as inaccurate as the assumption that they all do (I have XBMC on multiple platforms, including OE). This really isn't about platforms at any rate. There are (fewer) options for other platforms as well, the 3 I referenced were just the ones on the top of my head at the time as they are 3 of the more popular choices. Unfortunately, like most things *nix, the options and features are slimmer.

I can say that every OE user has XBMC and linux since it is XBMC + linux. Beyond that requiring windows will exclude a portion of the userbase therefore its not a solution that can be recommended. But its a point well taken.

(2012-11-21, 19:30)j114 Wrote: As far as the point about disk spin ups, I should point out that my "NAS" I am referencing is actually an 18 disk unRAID array hosted on my ESXi box which also hosts the Windows VM's that contains my "master" copies of XBMC and media scraper. I have my scraper configured to monitor my media collection and automatically process any new content. XBMC then scans for new content at regular intervals and adds my already processed content (so no Internet scraping of it's own).

Its an intersting idea but again far to specific to be a solution for everyone.

(2012-11-21, 19:30)j114 Wrote: For the thumbnails, If you update art on one client, then simply sync all of your cache directories to a single master cache directory. Maybe I am being thick this morning and just not seeing the cause for any excessive increase in drive spin ups using multiple cache directories on a single drive. I'm certainly not seeing it in my environment, but that's not to say that everyone has things configured the way I do either.

The ticky part is that syncing art needs syncing textures.db. This is possible but it would introduce a small boot delay (longer if the disk is spun down). But its definitely possible

(2012-11-21, 19:30)j114 Wrote: I call it a best practice to store all metadata and artwork alongside the actual media because it makes it far easier to migrate across platforms and frontends, as well as saving on Internet bandwidth consumption now and in the future. Obviously, the goal is to find a solution that works for you with your unique environment and parameters, so my "best practice" might not be yours.

Agreed. These are all points well taken. The point of this excecise for me was to understand how it all works now beyond the limited existing documentation and look to answering in advance the questions that the ever growing OE userbase moving from Eden to Frodo will ask.


But credit where credit is due you have given me an idea that may be a kludge for the die hard shared path sub users that were inevitably going to get. Will experiment and post back in a few days
Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#54
I just completed my first Frodo test upgrade with strange results.

I copied existing art cache and upgraded. XBMC booted, created the new mysql databse. When i tried to go to "movies" it prompted me to "re do art cache (paraphrase)".

XBMC went away for 15 minutes and completed.

However in textures i have < 15,000 entrys when i would expect over 200,000 in this cut down test. I seem to have no episode thumbnails but posters etc seem fine.

Two questions:

1. How did XBMC create these 14,500 entrys. There no way with my rubbish sub 1mbit link it grabbed them all in that timeframe (even though they all have URLs in textures which would suggets it did)
2. Why am I missing so many images
Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#55
I can't give you definitive answers to either of those, but I can tell you that I had almost the same experience when I first migrated to the nightlies. I had assumed that it was some bug in the build I was using and didn't think much of it. The main difference I experienced was that a lot of my movies were also missing most or all of their artwork. I did a full library (movies and TV) refresh and that seemed to correct the issue.
Reply
#56
Try the latest nightly build (which is beta1 + bug fixes leading to beta2). There's a bug that was fixed that can cause some artwork updating issues where not all of the artwork shows up for the library.
Reply
#57
(2012-11-25, 03:20)Ned Scott Wrote: Try the latest nightly build (which is beta1 + bug fixes leading to beta2). There's a bug that was fixed that can cause some artwork updating issues where not all of the artwork shows up for the library.

Thanks.

I have just completed a second test install using:

Starting XBMC (12.0-BETA1 Git:5417cfb), Platform: Linux (OpenELEC - Version: devel-20121125214720-r12586, 3.6.7 x86_64). Built on Nov 25 2012

Unlike last time i removed all Eden thumbnails from the cache before upgrading (As Frodo should regrab them all anyway). I noticed on the last test the 15,000 images Frodo upgraded added to the old Eden style cache i.e. all Eden art remained.

Upgrade completed successfully database wise but this time I have no thumbnails at all. This might not be a bad thing but I would need to inspire Frodo to re-grab them all and I do not know how to do this.

Thoughts?

Update: I deleted Textures13.db and rebooted. Frodo created a new Textures13.db but it is empty. Move and tv views are complete text wise but zero art of any sort

Update 2: I can manually refresh a single movie resulting in fanart, poster and actor thumbs (33 total). I believe that proves most things are setup correctly?

Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#58
I upgraded to:

XBMC (12.0-BETA1 Git:f14f5a5), Platform: Linux (OpenELEC - Version: devel-20121126192516-r12604, 3.6.7 x86_64). Built on Nov 26 2012

with the same no existing art set-up. This upgraded the dbase version to 74. I then removed Textures13.db and rebooted.

As expected Textures13.db was recreated. What is interesting is that in the previous version i manually refreshed one movie resulting in XBMC caching 33 images (mostly actors). However in the logs after the upgrade i can see XBMC grabbing the poster and fanart from this movie again.

So I can now see the automagic grabbing of art but ONLY for movies I have previously manually refreshed after the Frodo upgrade. ALL other movies and tv shows with no art do nothing.

What am I missing? Is this expected behaviour?

Update: I started over yet again. Randomly things seem to be going better this time. It is certainly taking alot longer. The only real difference this time is that I ran the java thumbnail cleanup script first removing 2GB of redundant Eden art.
Update 2: This time it completed after a period of severe CPU load load average: 8.87, 7.79, 5.03. All posters and fanart look good but yet again no episode thumbnails. I need an assist here I have nothing else left to try
Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply
#59
Strange. Just to verify that I remembered correctly, I deleted my textures13.db and my thumbnail cache on one of my SQL client machines. Upon restart of XBMC, it began rebuilding the artwork as soon as I entered each library. This did take a little while to complete (roughly 20 minutes-- I was multi-tasking and not watching the clock) with moderate to high cpu usage, but once finished all of my artwork is in place. Including episode thumbs.

I'm sure a complete refresh of your TV library would pull the images again, but that's obviously not an ideal solution. If you go into a TV show at episode level and let it sit there for a couple of minutes, is it still not pulling thumbnails at all? By chance, has your content been reset to none somehow during the upgrade process?
Reply
#60
(2012-11-28, 08:36)j114 Wrote: ...
I'm sure a complete refresh of your TV library would pull the images again, but that's obviously not an ideal solution. If you go into a TV show at episode level and let it sit there for a couple of minutes, is it still not pulling thumbnails at all?...

Nope I can let it sit there for an hour and it does nothing

(2012-11-28, 08:36)j114 Wrote: ...
has your content been reset to none somehow during the upgrade process?

Confirmed scraper still set on all sources

I manually refreshed one show and it grabbed a bunch of art. However even then it only filled one season of thumbs. This is interesting as this particular show has HD and SD episodes in two different locations/NAS.

So what i see with mixed path sources HD/SD:

In Library no fanart or top level poster
In Library no poster or ep thumbs for SD source
In Library poster and ep thumbs for HD source
In file view SD source shows no art at all
In file view HD source HAS fanart, poster and thumbs


For a single path source HD or SD i get all art on refresh.

So i went and found another mixed HD/SD set that has two sources and this time all the SD has art on refresh and HD doesn't (i.e. exact opposite of before).

To me this seems like a bug with multi path sources

Having problems getting your TV shows recognized?

Try my extra TV show matching REGEX here
Reply

Logout Mark Read Team Forum Stats Members Help
FRODO Thumbnail sync storage0