Frodo - Shared Library Thumbnail Updates & Mulitple PCs

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #16
Pr Sinister: I have a large collection but I don't store the images in the directories like you any more. I used to do that and use the Ember Media Manager (I think that was what it was called) to create the JPG and .NFO file for each movie. Now I let XBMC do it on its own and as long as the movie naming convention - name (year) is right there are almost no issues.

On the odd chance it gets it wrong I update it through the XBMC interface. This works really well and saves me the extra step of manual processing.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,228
Joined: Oct 2003
Reputation: 177
Post: #17
greenman: The URL is the URL to wherever the art came from.

You're welcome to copy the cache between machines to your hearts content. The problem with the old one is that it wasn't portable, as everything depended on the client's URL to the video file. This is why you couldn't have video files local to machine A but addressed via smb:// from machine B (shared from A). Further, we didn't even know where the art came from originally. Now, we know all that.

The reason we keep things local is because it's significantly faster for the vast majority of client machines. Not all machines, mind (e.g. if you have a gigabit network with the art on an SSD on the other end or something) but in those cases then pathsubstitution can still be used, it's just not particularly recommended in a multi-client scenario (multiple writes to the same file are possible), though will likely be perfectly safe.

In the future, it's likely that some of the caching will be optional for these types of cases. This is relatively easy for us to add now that the texturecache is standardized.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
greenman Offline
Junior Member
Posts: 11
Joined: Aug 2007
Reputation: 0
Post: #18
jmarshall: Thanks for the clarification that pathsubstitution can still be used.

I'll certainly try out the new system however it's good to know I can revert back to my current setup with Frodo.

There is very little lag with the centrally stored thumbnails on a NAS over a gigabit network.
find quote
Pr.Sinister Online
Fan
Posts: 663
Joined: Sep 2007
Reputation: 14
Location: Montreal, Canada
Post: #19
(2013-02-01 05:23)jmarshall Wrote:  Pr. Sinister: No way in hell it should take that long. The textures update for instance is essentially: "drop everything and change a couple tables" so anything more than a few seconds is way too long. Same for the others.

The music scan could well be about right if you have a really, really large collection - essentially it's a complete (local) rescan of your music. I've only got a small collection (some 5k songs) and it takes about 5-10 minutes (over wifi).

The way i have XBMC Setup is like so :

SERVERS

CNS-TVSERVER : Windows 7, Wired Gigabit, SMB Shares, Local Art & .nfos, No XBMC
CNS-MOVIESERVER : Windows 7, Wired Gigabit, SMB Shares, Local Art & .nfos, No XBMC
CNS-FILESERVER (Music) : Windows 7, Wired Gigabit, SMB Shares, Local Art, XBMC, MySQL, Path Substituted Thumbnails folder

CLIENTS

CNS-Bedroom : Windows 7, Wired Gigabit, XBMC, Using CNS-FILESERVER's Path Substituted Thumbnails folder
CNS-Livingroom : Windows 7, Wired Gigabit, XBMC, Using CNS-FILESERVER's Path Substituted Thumbnails folder
CNS-Basement : Windows 7, Wired Gigabit, XBMC, Using CNS-FILESERVER's Path Substituted Thumbnails folder
CNS-Kitchen : Windows 7, Wired Gigabit, XBMC, Using CNS-FILESERVER's Path Substituted Thumbnails folder
Raspberry Pi : OpenElec XBMC, Wired Gigabit, Using CNS-FILESERVER's Path Substituted Thumbnails folder

CNS-FILESERVER is never used to watch or browse the library. Only the clients are used to play and browse the media.

When i ran the upgrade, i ran it on CNS-FILESERVER.

I noticed the log also found a lot of thumbnails that were not cached... Mostly actors... Probably because i rarely browse by actor since there are way too many in my database...

Here is a link to my log in non-debug mode if you want to take a look. I expected it to take quite some time because even when i reload my library from scratch (with all local artwork and nfos), it usually takes more than one full day.

------------------------------------------
[4 XBMC Clients + 3 Norco RPC-4224 Media Servers w/74 TB HDD Space]
If my post helped, thank me by adding to my reputation! :)
find quote
Sethvir Offline
Junior Member
Posts: 4
Joined: Feb 2013
Reputation: 0
Post: #20
Not to beat a dead horse, but IMHO it seems the crux of the question has not really been addressed.

Consider the following (probably fairly typical) setup under Eden:

1. NAS containing shares for movies, tv, and music.
2. NAS also runs a central MySQL database.
3. NAS also contains a share for thumbnails/fanart, referenced by the same path substitution on each XBMC installation.
4. Several XBMC installations which are basically "cloned" by copying advancedsetting.xml and sources.xml.
5. Note that ALL shared content is on the NAS. This is the only "always-on" machine in the house and sharing files from any other machine is not acceptable.

One of my XBMC installations runs on my desktop windows PC and I primarily use this installation for administration (e.g. scanning in new movies/tv-episodes).

Most of my thumbnails and fanarts are scraped automatically, however occasionally I need to obtain an image from another source. This might be from somewhere like google images, or I might scan an image directly, or I might even create my own. In any case, I end up with an image on my desktop PC.
I then start XBMC on my desktop PC and select the image by browsing to the local image. XBMC then automatically copies this up to the NAS by virtue of the path substitution, and as long as all XBMC installations use the same path substitution, they all immediately display the new image.
I typically then delete the local image from my desktop PC.

Now, under Frodo, I can't see how this use case can work...
The scraped images are more or less OK, of course (although I concur that the multiple downloads are less than ideal, and I find the assumption of immutability for URLs on the web to be less than convincing).
The issue is the local files. What "URL" will be stored in the library for these images? They have been accessed from a local drive on my desktop PC, so there is no real URL, and certainly not one that would be accessible from my other XBMC installations. Even it it were, I do not want to share content from this PC, but only from my NAS (see point 5 above).

It has been suggested in some previous replies that path substitution will still work, but I can't see how. XBMC will want to store "where the file came from" (i.e. its URL) in the library and for local files on my desktop PC, that information is useless.


The only solution that I can see is to create my own repository for non-scraped images as a share on my NAS. All XBMC installations would have a source path for this share and before using such an image, I would have to copy it to the share and then browse for it in XBMC in that location. Presumably, the URL stored in the library would then be usable on all of my XBMC installations.
I actually do something like this now for movieset thumbs and fanart as these are not dumped on an export and so would be lost if I had to rebuild my library. (Of course, there are no URLs being stored for these images under Eden).
I suspect this will work for individual thumbs and fanart, but it seems to me I am now having to do more work to maintain an extra repository of non-standard images - something which XBMC basically did for me automatically under Eden.

I suppose rather than creating a separate share, I could simply store the images with the media files (e.g. <filename>.tbn), and have XBMC scrape the local files...


BTW greenman,

I am not sure why you are having each of your XBMCs run a library update on startup?
If you are using MySQL, then you should just need to do a single update from one machine when you add new content. All XBMC's will then automatically see the new content as they see the same library.
Personally, I just scan for new content in a particular directory rather than a full update.
Unless the new content is being added by some sort of automatic process?
(This post was last modified: 2013-02-04 08:09 by Sethvir.)
find quote
Pr.Sinister Online
Fan
Posts: 663
Joined: Sep 2007
Reputation: 14
Location: Montreal, Canada
Post: #21
(2013-02-04 08:00)Sethvir Wrote:  It has been suggested in some previous replies that path substitution will still work, but I can't see how. XBMC will want to store "where the file came from" (i.e. its URL) in the library and for local files on my desktop PC, that information is useless.

The URL that is stored will only be accessed if somehow the cached version is no longer available. It goes to the original image, makes a copy of it in the thumbnails folder, and stores the original location in case it needs to fetch it again. The first thing XBMC does when having to display an image is it checks if a cached version exists. If not, it tries to go to the URL and if that doesn't work then it will generate its own thumb from the media depending on your settings or skin.

Quote:The only solution that I can see is to create my own repository for non-scraped images as a share on my NAS. All XBMC installations would have a source path for this share and before using such an image, I would have to copy it to the share and then browse for it in XBMC in that location. Presumably, the URL stored in the library would then be usable on all of my XBMC installations.

(snip)

I suppose rather than creating a separate share, I could simply store the images with the media files (e.g. <filename>.tbn), and have XBMC scrape the local files...

Out of the 2, the latter is definitely what i suggest. It is much easier to manage and you know you will never have any broken URLs. Actually, when you export your library to multiple files, this is how XBMC stores your thumbs. It obviously means that it is the best way for XBMC to keep your library and art in its original state.

-Pr.

------------------------------------------
[4 XBMC Clients + 3 Norco RPC-4224 Media Servers w/74 TB HDD Space]
If my post helped, thank me by adding to my reputation! :)
find quote
Kib Offline
Team-Kodi Server Dude
Posts: 3,399
Joined: Jan 2010
Reputation: 53
Location: NL
Post: #22
Combine the above with the Artwork Downloader addon and use its option 'Use local files' and it is a great combination in my opinion.

Neon skin - v3.0.8 now available for XBMC 13.x (Gotham)
find quote
Sethvir Offline
Junior Member
Posts: 4
Joined: Feb 2013
Reputation: 0
Post: #23
(2013-02-04 17:47)Pr.Sinister Wrote:  The URL that is stored will only be accessed if somehow the cached version is no longer available. It goes to the original image, makes a copy of it in the thumbnails folder, and stores the original location in case it needs to fetch it again. The first thing XBMC does when having to display an image is it checks if a cached version exists. If not, it tries to go to the URL and if that doesn't work then it will generate its own thumb from the media depending on your settings or skin.

ICBW, but I was under the impression (from other threads) that each Frodo XBMC client has a local (theoretically unshareable) SQLLite thumbs database (texture.db) which now controls all artwork caching, and that as a result, each "secondary" XBMC client would attempt to re-download the artwork for new media (rather than relying on the existence of the file in the shared cache as for Eden). In that case, the URL _would_ be used (but would fail for image files browsed locally on my "primary" client), and those "secondary" clients would never see the cached artwork. BTW - if this is true, it also renders the concept of a shared cache fairly pointless, and the only remaining purpose for path substitution seems to be to relocate the cache (e.g. to save space on a limited-storage device).

Can you confirm that the presence of the cached file is actually still taken into account under Frodo?

(2013-02-04 17:47)Pr.Sinister Wrote:  Out of the 2, the latter is definitely what i suggest. It is much easier to manage and you know you will never have any broken URLs. Actually, when you export your library to multiple files, this is how XBMC stores your thumbs. It obviously means that it is the best way for XBMC to keep your library and art in its original state.

Yep - I already export to individual files, but only infrequently. Of course under Eden, these exported artwork files are only ever used (in my environment) when explicitly refreshing or reloading media. Just to be clear, the difference with what I am suggesting for Frodo would be that locally sourced (rather than scraped) artwork would also be _directly_ referenced as a source location by URLs stored in the shared MySQL database, and so for new media, each client would be dynamically reloading its thumbnail cache (whether shared or individual from my understanding) from the media folders (or other shared directory).

This is, of course, not the same thing as updating the library on each client at startup, which I absolutely do not want to do, as it would be a backward step from the seamless dynamic update I currently get under Eden.

Can you see any issues with the using the image files in the media folders in this manner? (i.e. as the "source" URL).
(This post was last modified: 2013-02-05 04:38 by Sethvir.)
find quote
Ned Scott Offline
Team-Kodi Wiki Guy
Posts: 20,352
Joined: Jan 2011
Reputation: 258
Location: Arizona, USA
Post: #24
Updating the library on each client is not a step backwards or something that will negatively impact performance. All XBMC would do is go to check to see if files have changed for a library update, see that nothing has changed, and thus not actually do any update.

XBMC will use a local URL path for images if that's how you added the image. If you manually selected it from say C:\Computer\desktop or whatever, then every XBMC box will look in that location. If you select it from the network share where the exported image would be, XBMC will store that network path and each XBMC box will find it.

You can make easy links to the XBMC wiki using double brackets around common XBMC words: debug log = debug log, Video library = Video library, SMB = SMB , userdata = userdata, etc
find quote
Sethvir Offline
Junior Member
Posts: 4
Joined: Feb 2013
Reputation: 0
Post: #25
I appreciate the replies. Thanks.

(2013-02-05 04:23)Ned Scott Wrote:  Updating the library on each client is not a step backwards or something that will negatively impact performance. All XBMC would do is go to check to see if files have changed for a library update, see that nothing has changed, and thus not actually do any update.

The thing is that I don't currently need to do anything at all on the "secondary" clients in order to see new media and/or artwork. Having to do an update to see the new artwork would mean actually having to restart xbmc to initiate the update (or manually initiating an update on each machine). Most of my XBMCs are not regularly restarted. I was just wanting to make the point that I want the process under Frodo to be as seamless as it was under Eden using a shared cache.

(2013-02-05 04:23)Ned Scott Wrote:  XBMC will use a local URL path for images if that's how you added the image. If you manually selected it from say C:\Computer\desktop or whatever, then every XBMC box will look in that location. If you select it from the network share where the exported image would be, XBMC will store that network path and each XBMC box will find it.

Great. That is what I was expecting/hoping.
In that case, I occurs to me that it might be a good idea in upgrading to Frodo to have all URLs for my current artwork referencing my exported artwork files as source URLs. Moving forward, new scraped artwork would reference web URLs, but I am happy with that, and I expect I could periodically export and update URLs to reference local files to protect against future broken links.

The questions that raises then are:
1. How does the Frodo upgrade initially populate the URLs for existing artwork? (it is not stored for Eden, right?)
2. What is the best way to update all artwork URLs to reference local files? Full library reload from exports? Will that store the URLs of the local image files?
(This post was last modified: 2013-02-05 05:01 by Sethvir.)
find quote
Sethvir Offline
Junior Member
Posts: 4
Joined: Feb 2013
Reputation: 0
Post: #26
Hey Ned,

Looks like you have already answered my last question elsewhere:

(2012-12-29 05:36)Ned Scott Wrote:  If you export the library as multiple files from the working XBMC box, then XBMC will place the artwork along side your media. From there you can then rescan your library (should be identical), and XBMC will use a network URL for the images that are stored next to the media files.

So I guess that answers my question - export and rescan should handle all of the existing artwork, and in future any non-scraped artwork should be placed into the media folder first and selected from there.
find quote
Ned Scott Offline
Team-Kodi Wiki Guy
Posts: 20,352
Joined: Jan 2011
Reputation: 258
Location: Arizona, USA
Post: #27
Yeah, I'm thinking using the multiple file export should be the standard advice now for MySQL, for both creating a new MySQL set up and for updating. It's the safest method and should have the least issues. I've started to update the guide to reflect that. I've been reluctant to make this the main advice in the past since I normally don't like having NFO files next to media, but it's worth it for the easier set up and as a fail-safe backup of the library.

You can make easy links to the XBMC wiki using double brackets around common XBMC words: debug log = debug log, Video library = Video library, SMB = SMB , userdata = userdata, etc
find quote
aptalca Offline
The Dude
Posts: 912
Joined: Sep 2009
Reputation: 25
Post: #28
These are excellent questions

I tried the artwork downloader with the use local files option (for only clearart to test), also selected overwrite

The effect was that my database url's were modified to show smb paths, which is what I wanted.

But I cannot do the same with fanart, posters, etc because I do not want my existing art to be changed (overwritten)

If I export to separate files, then rescan, wouldn't that mess up the watched status (I don't think it is stored in the nfo's)

I think it would be the best if we had an option to also update the urls in the database with the local ones during an export to separate files. It would be even better to have an option for the scrapers, so that during library updates xbmc would not only download the artwork to cache, but also save it with the media, and use that as the url in the database for other clients. That way I wouldn't have to deal with constant exports.

Every time I set up a new computer, it takes ages to download the artwork from the web. It would be much much faster if the urls pointed to smb shares where the media resides
(This post was last modified: 2013-02-06 20:06 by aptalca.)
find quote
spchtr Offline
Junior Member
Posts: 25
Joined: Dec 2011
Reputation: 0
Post: #29
I seem to be experiencing the same problem.

I am attempting to find a solution.

I think the problem may be the Thumbnail.db file in /portable_data/userdata/Databases/.

I am going to copy that file to the other systems after adding a movie via my most used system.

My theory is the Thumbnail.db file stores the hashed name of the [thumbnail].jpg and the directory inside the Thumbnails folder.

My Thumbnails folder has been redirected to an smb share that is read write /w username and password, and has been redirected on all of my machines.

The [thumbnail].jpg file should have been added to the shared folder, but the reference in Thumbnails.db is not updated on the other machines.

By syncing this small file, I hope to trick the other machines into accepting the [thumbnail].jpg

I'll let you all know if it works.

Personally I'd prefer it if xbmc could uuencode the thumbnail.jpg or something similar and put it inside the mysql database. I know access databases have this functionality, and in this situation that would have been preferable.

Barring that, I wish advancedsettings.xml had a substitution or redirect for the <thunbnailsdatabase>.
(This post was last modified: 2014-02-26 06:10 by spchtr.)
find quote
spchtr Offline
Junior Member
Posts: 25
Joined: Dec 2011
Reputation: 0
Post: #30
Ok, turns out I got it working, One of the problems was I was using the netbios name for the smb shares instead of the ip address of the mysql server and smb share of the Thumbnails folder.

Once those were corrected, if I add a movie on one system, I still have to copy the userdata/Databases folder to each machine from that one system where I added the movie to the library to get the thumbnails to show up.

I'm still hoping they add the ability to redirect the thumbnail.db into the mysql server to have that centralized as well.
find quote