[rubpa]

Hi All! I'm trying to solve some issues that I observe in the Pictures module of Kodi. As a start, I'm looking into thumbnails generation taking forever with large folders (500+).

Here's my understanding so far:

• BackgroundInfoLoader is used to create a JobQueue for thumbnails creation
  
• The JobQueue uses JobManager to execute the actual TextureCacheJob
  

In one of my tests with a 500 images folder, only about 33 thumbnails were created even after few minutes of leaving Kodi idle. In the logs, I observed these:

2022-10-25 15:41:01.595 T:15748    info <general>: Skipped 19 duplicate messages..
2022-10-25 15:41:04.578 T:15748    info <general>: Skipped 5 duplicate messages..
2022-10-25 15:41:08.781 T:3328     info <general>: Skipped 2 duplicate messages..
2022-10-25 15:41:19.737 T:15748    info <general>: Skipped 19 duplicate messages..
2022-10-25 15:41:28.020 T:17356    info <general>: Skipped 16 duplicate messages..
2022-10-25 15:41:34.213 T:10244    info <general>: Skipped 2 duplicate messages..
2022-10-25 15:41:45.378 T:15748    info <general>: Skipped 19 duplicate messages..

I think this is causing the thumbnail generation to be skipped. However, I'm unable to find these messages in the actual code. Any pointers appreciated!

[wsnipex]

turn on debug logging, since this doesn't show the actual skipped log line.

[rubpa]

I did have debug turned on + additional logs that I had added to understand the flow. Here's the full log : https://paste.kodi.tv/havehixedo.kodi (logs prefixed with xx were added by me)

The folder c-fivehundred had 500 photos and as you can see only about 30 thumbnails were created. What I'm not able to figure out is where these "skipped duplicate" messages are coming from.

[wsnipex]

this is async / background media scanning, there are tons of jobs put into the queue:

2022-10-25 15:41:31.550 T:13900   debug <general>: xx CTextureCache::BackgroundCacheImage AddJob image://C%3a%5cgit-ws%5ctest-photos%5cc-fivehundred%5ca0392.JPG/transform?size=thumb
2022-10-25 15:41:31.555 T:13900   debug <general>: xx CTextureCache::BackgroundCacheImage AddJob image://C%3a%5cgit-ws%5ctest-photos%5cc-fivehundred%5ca0393.JPG/transform?size=thumb
2022-10-25 15:41:31.560 T:13900   debug <general>: xx CTextureCache::BackgroundCacheImage AddJob image://C%3a%5cgit-ws%5ctest-photos%5cc-fivehundred%5ca0394.JPG/transform?size=thumb
...

later the job queue is periodically unpaused, so processing is started:
2022-10-25 15:44:16.317 T:15748 debug <general>: xx CJobManager::UnPauseJobs
2022-10-25 15:44:26.678 T:15748 info <general>: Skipped 19 duplicate messages..

If you want to figure out which log line has been skipped(the log line repeated x times), look at the thread id, it this case T:15748.
Just filter by thread id then the line immediately above is the skipped one.
In the case above: 19 Jobs have been unpaused.

edit: it looks like you shut kodi down before scanning was complete

[rubpa]

If you want to figure out which log line has been skipped(the log line repeated x times), look at the thread id, it this case T:15748.
Just filter by thread id then the line immediately above is the skipped one.
In the case above: 19 Jobs have been unpaused.

Ah! so the "skipped" message is by the logger indicating that it did not log repeated messages. I was wondering if it's the job queue skipping jobs.

edit: it looks like you shut kodi down before scanning was complete

Yes, I waited for 2 mins and then closed kodi. This replicates the problem I see on my HTPC with v19.4 - large folders seem to take forever to create thumbnails. I'll continue to look into it.

[rubpa]

Phew! The fun of multithreaded debugging.
 
I was able to reproduce the issue with my test folder of 500 images quite often and here's what I have found.
 
First the issue as observed by the end user/me: when I open a Pictures folder I expect Kodi to create/load the thumbnails. In the recent years, large folders have been a problem. Sometimes all thumbnails are created/cached fine. Sometimes, they don't load even after leaving the folder open for 10-15 mins. Scrolling through the list creates the thumbnails of visible images but this method is quite painful for folders with 1000+ images. Also, this causes Kodi crashes too.
 
The issue as I understand after debugging:
CPictureThumbLoader uses CTextureCache::BackgroundCacheImage() to create a CJobQueue (CTextureCache is a CJobQueue) of all images in the folder for which thumbnails are required. Only the 1st job is actually added to CJobManager as the CJobQueue is initialized with jobsAtOnce=1. When the job is processed by CJobManager, CJobQueue::OnJobComplete() is called which queues the next job.
Sometimes, the thumbnails stop loading because CJobQueue::OnJobComplete() is not called and then the CJobManager is no longer fed with new jobs from the CTextureCache queue.
 
Why is CJobQueue::OnJobComplete() not called sometimes? Because CJobManager::CancelJob() was called for some job id which sets the CWorkItem's m_callback to NULL.
 
Why is CJobManager::CancelJob() called? Because CTextureCacheJob:: DoWork() determined that the job should be cancelled.
 
And why? Because CTextureCache::OnJobProgress() determined that the same url is being processed currently.
 
And why again? Because CGUILargeTextureManager::QueueImage() had queued the same image for thumbnail creation.
 
So, there is a race between CGUILargeTextureManager & CTextureCache and in certain conditions, the CTextureCache stops feeding the CJobManager.
 
Let me know any comments. Shall I propose a fix via a PR?

[wsnipex]

the rabbit hole is pretty deep here, nice debugging!
A PR would be much appreciated

[bossanova808]

Just a thanks for this - larger folders of images with Kodi is thoroughly broken, has been for a while, and I for one will be super appreciative to see it fixed!

[bossanova808]

Did this ever make it to the PR stage?  I am still seeing issues with very slow folders of images when I browse them in Thumbnail mode.

It may just be the Odroid N2+ GPU not having much ram, I suppose (or to do with the scrolling labels issue I saw  a fix go through for) - but it pretty much breaks the Kodi picture functionality for me, so I'm trying to narrow it down!

[rubpa]

Yes, it did get to the PR stage and was merged - https://github.com/xbmc/xbmc/pull/22082

The fix improved the "reliability" of the thumbnail generation, not the speed. I recently updated my HTPC to 20.1 and a folder of 2000+ photos got all the thumbnails in the first try - which was impossible earlier. However, it did take a looong time since this runs as a background single thread job - there was a specific discussion to not improve performance by increasing threads in the interest of low RAM devices.

[bossanova808]

Ok that's good to know. 

My issues is that the thumbnails do appear (not too sluggishly really) - but the UI just competely locks up and for a minute or more I can't move - all I can really do to get back into an ok state is exit that window...

I do therefore wonder if it is about the scrolling labels issue, more so than the pictures.  But there's definitely _something_ significantly wrong.

[Karellen]

IIRC @CrystalP fixed some Picture issues recently.... https://github.com/xbmc/xbmc/pull/23268
Don't know if it is the same issue you are reporting, though.

[bossanova808]

I don't think that is related (no corruption for me).

I do think it's possible this IS related - it certainly wouldn't help, as all my images have long labels and therefore scroll...
https://github.com/xbmc/xbmc/pull/23233

Do you happen to know if that one is/could be backported??

With these little systems, it's a bit hard to try new versions etc than with PCs and whatnot...so I don't have any omega testbeds around.

[Karellen]

Do you happen to know if that one is/could be backported??

No idea. You would be best asking @CrystalP
Maybe ask in that PR?

[bossanova808]

Have done so - thanks!