[tinybutstrong]

Someone know if something changed for the advancedsettings.xml option to cleaning after to update the library on Gotham alpha 10/Nightly/any?

I have <cleanonupdate>true</cleanonupdate> for videos/music/.

It is taking long time to do the job, like 100x more than current Frodo.

Thanks.

[tinybutstrong]

Please some dev to comment upon my question?

Tried to run the cleanup now (from submenu this time), 45 minutes and still running..

[Ned Scott]

Can you get us a debug log (wiki) of when this happens?

[MrDVD]

I have exact the same problem, and i think it started at the same time as xbmc switched from database version xbmc_video76 to database version xbmc_video77.

Here is the part from the log where the canner is active.

9:56:40 T:6272 NOTICE: Starting XBMC (13.0-ALPHA11 Git:20131214-e13564c). Platform: x86 Win32 32-bit
...
09:58:27 T:6272 DEBUG: CApplication::OnKey: return (0xf00d) pressed, action is Select
09:58:27 T:1680 NOTICE: Thread VideoInfoScanner start, auto delete: false
09:58:27 T:6272 DEBUG: ------ Window Init (DialogExtendedProgressBar.xml) ------
09:58:27 T:1680 NOTICE: VideoInfoScanner: Starting scan ..
09:58:27 T:1680 DEBUG: CAnnouncementManager - Announcement: OnScanStarted from xbmc
09:58:27 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnScanStarted
...
09:58:27 T:1680 DEBUG: VideoInfoScanner: No (new) information was found in dir smb://UBUNTU/Serien/Fairly Legal/
...
09:58:28 T:1680 DEBUG: VideoInfoScanner: Scanning dir 'smb://UBUNTU/hdtv/HDTV/' as not in the database
09:58:28 T:1680 DEBUG: VideoInfoScanner: No (new) information was found in dir smb://UBUNTU/hdtv/HDTV/
...
09:58:29 T:1680 DEBUG: VideoInfoScanner: Skipping dir 'smb://UBUNTU/hdtv/HDTV/Der Junge und der Wolf (2009)/' due to no change
09:58:29 T:1680 DEBUG: VideoInfoScanner: Skipping dir 'smb://UBUNTU/hdtv/HDTV/Der Junge und der Wolf (2009)/extrafanart/' due to no change (fasthash)
..
09:58:29 T:1680 NOTICE: CVideoDatabase::CleanDatabase: Starting videodatabase cleanup ..
09:58:29 T:1680 DEBUG: CAnnouncementManager - Announcement: OnCleanStarted from xbmc
09:58:29 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnCleanStarted
09:58:29 T:1680 DEBUG: Mysql Start transaction
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning paths that don't exist and have content set...
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning tvshow table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM tvshow WHERE NOT EXISTS (SELECT 1 FROM tvshowlinkpath WHERE tvshowlinkpath.idShow = tvshow.idShow)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning path table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM path WHERE (strContent IS NULL OR strContent = '') AND (strSettings IS NULL OR strSettings = '') and (strHash IS NULL OR strHash = '') AND (exclude IS NULL OR exclude != 1) AND NOT EXISTS (SELECT 1 FROM files WHERE files.idPath = path.idPath) AND NOT EXISTS (SELECT 1 FROM tvshowlinkpath WHERE tvshowlinkpath.idPath = path.idPath) AND NOT EXISTS (SELECT 1 FROM movie WHERE movie.c23 = path.idPath) AND NOT EXISTS (SELECT 1 FROM tvshow WHERE tvshow.c17 = path.idPath) AND NOT EXISTS (SELECT 1 FROM episode WHERE episode.c19 = path.idPath) AND NOT EXISTS (SELECT 1 FROM musicvideo WHERE musicvideo.c14 = path.idPath)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning genre table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM genre WHERE NOT EXISTS (SELECT 1 FROM genrelinkmovie WHERE genrelinkmovie.idGenre = genre.idGenre) AND NOT EXISTS (SELECT 1 FROM genrelinktvshow WHERE genrelinktvshow.idGenre = genre.idGenre) AND NOT EXISTS (SELECT 1 FROM genrelinkmusicvideo WHERE genrelinkmusicvideo.idGenre = genre.idGenre)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning country table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM country WHERE NOT EXISTS (SELECT 1 FROM countrylinkmovie WHERE countrylinkmovie.idCountry = country.idCountry)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning actor table of actors, directors and writers
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM actors WHERE NOT EXISTS (SELECT 1 FROM actorlinkmovie WHERE actorlinkmovie.idActor = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinkmovie WHERE directorlinkmovie.idDirector = actors.idActor) AND NOT EXISTS (SELECT 1 FROM writerlinkmovie WHERE writerlinkmovie.idWriter = actors.idActor) AND NOT EXISTS (SELECT 1 FROM actorlinktvshow WHERE actorlinktvshow.idActor = actors.idActor) AND NOT EXISTS (SELECT 1 FROM actorlinkepisode WHERE actorlinkepisode.idActor = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinktvshow WHERE directorlinktvshow.idDirector = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinkepisode WHERE directorlinkepisode.idDirector = actors.idActor) AND NOT EXISTS (SELECT 1 FROM writerlinkepisode WHERE writerlinkepisode.idWriter = actors.idActor) AND NOT EXISTS (SELECT 1 FROM artistlinkmusicvideo WHERE artistlinkmusicvideo.idArtist = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinkmusicvideo WHERE directorlinkmusicvideo.idDirector = actors.idActor)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning studio table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM studio WHERE NOT EXISTS (SELECT 1 FROM studiolinkmovie WHERE studiolinkmovie.idStudio = studio.idStudio) AND NOT EXISTS (SELECT 1 FROM studiolinkmusicvideo WHERE studiolinkmusicvideo.idStudio = studio.idStudio) AND NOT EXISTS (SELECT 1 FROM studiolinktvshow WHERE studiolinktvshow.idStudio = studio.idStudio)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning set table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM sets WHERE NOT EXISTS (SELECT 1 FROM movie WHERE movie.idSet = sets.idSet)
09:58:31 T:1680 DEBUG: Mysql commit transaction
09:58:31 T:1680 NOTICE: CVideoDatabase::CleanDatabase: Cleaning videodatabase done. Operation took 00:02
09:58:31 T:1680 DEBUG: CAnnouncementManager - Announcement: OnCleanFinished from xbmc
09:58:31 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnCleanFinished
09:58:31 T:1680 NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:04
09:58:31 T:1680 DEBUG: CAnnouncementManager - Announcement: OnScanFinished from xbmc
09:58:31 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnScanFinished
09:58:31 T:1680 DEBUG: Thread VideoInfoScanner 1680 terminating
09:58:31 T:6272 DEBUG: CGUIMediaWindow::GetDirectory ()

Duno if this is the problem "NOTICE: Thread VideoInfoScanner start, auto delete: false" and how to change this but "<cleanonupdate>true</cleanonupdate>" is still inside my advancedsettings.xml

Edit: Video77 isnt the problem as with XBMCSetup-20131118-0018656-master.exe it works an there was also already Video77.

Edit2: something between XBMCSetup-20131201-73a1b58-master.exe and XBMCSetup-20131201-1f67afd-master.exe went wrong and make the error as with 73a1b58 all works and with 1f67afd from the same day didnt work anymore.

[tinybutstrong]

Can you get us a debug log (wiki) of when this happens?

Ok, can you check this debug log bellow:

http://xbmclogs.com/show.php?id=98341

took 40+ minutes.

Thanks.

[Ned Scott]

Ouch. I could easily be missing something, but I looked at the log for a while and wasn't able to see anything obvious. I do know of one semi-major change involving cleanonupdate, which is new for the December nightlies, and have asked if it might be related to this issue: http://forum.xbmc.org/showthread.php?tid...pid1575736

[Montellese]

I have exact the same problem, and i think it started at the same time as xbmc switched from database version xbmc_video76 to database version xbmc_video77.

Here is the part from the log where the canner is active.

9:56:40 T:6272 NOTICE: Starting XBMC (13.0-ALPHA11 Git:20131214-e13564c). Platform: x86 Win32 32-bit
...
09:58:27 T:6272 DEBUG: CApplication::OnKey: return (0xf00d) pressed, action is Select
09:58:27 T:1680 NOTICE: Thread VideoInfoScanner start, auto delete: false
09:58:27 T:6272 DEBUG: ------ Window Init (DialogExtendedProgressBar.xml) ------
09:58:27 T:1680 NOTICE: VideoInfoScanner: Starting scan ..
09:58:27 T:1680 DEBUG: CAnnouncementManager - Announcement: OnScanStarted from xbmc
09:58:27 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnScanStarted
...
09:58:27 T:1680 DEBUG: VideoInfoScanner: No (new) information was found in dir smb://UBUNTU/Serien/Fairly Legal/
...
09:58:28 T:1680 DEBUG: VideoInfoScanner: Scanning dir 'smb://UBUNTU/hdtv/HDTV/' as not in the database
09:58:28 T:1680 DEBUG: VideoInfoScanner: No (new) information was found in dir smb://UBUNTU/hdtv/HDTV/
...
09:58:29 T:1680 DEBUG: VideoInfoScanner: Skipping dir 'smb://UBUNTU/hdtv/HDTV/Der Junge und der Wolf (2009)/' due to no change
09:58:29 T:1680 DEBUG: VideoInfoScanner: Skipping dir 'smb://UBUNTU/hdtv/HDTV/Der Junge und der Wolf (2009)/extrafanart/' due to no change (fasthash)
..
09:58:29 T:1680 NOTICE: CVideoDatabase::CleanDatabase: Starting videodatabase cleanup ..
09:58:29 T:1680 DEBUG: CAnnouncementManager - Announcement: OnCleanStarted from xbmc
09:58:29 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnCleanStarted
09:58:29 T:1680 DEBUG: Mysql Start transaction
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning paths that don't exist and have content set...
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning tvshow table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM tvshow WHERE NOT EXISTS (SELECT 1 FROM tvshowlinkpath WHERE tvshowlinkpath.idShow = tvshow.idShow)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning path table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM path WHERE (strContent IS NULL OR strContent = '') AND (strSettings IS NULL OR strSettings = '') and (strHash IS NULL OR strHash = '') AND (exclude IS NULL OR exclude != 1) AND NOT EXISTS (SELECT 1 FROM files WHERE files.idPath = path.idPath) AND NOT EXISTS (SELECT 1 FROM tvshowlinkpath WHERE tvshowlinkpath.idPath = path.idPath) AND NOT EXISTS (SELECT 1 FROM movie WHERE movie.c23 = path.idPath) AND NOT EXISTS (SELECT 1 FROM tvshow WHERE tvshow.c17 = path.idPath) AND NOT EXISTS (SELECT 1 FROM episode WHERE episode.c19 = path.idPath) AND NOT EXISTS (SELECT 1 FROM musicvideo WHERE musicvideo.c14 = path.idPath)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning genre table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM genre WHERE NOT EXISTS (SELECT 1 FROM genrelinkmovie WHERE genrelinkmovie.idGenre = genre.idGenre) AND NOT EXISTS (SELECT 1 FROM genrelinktvshow WHERE genrelinktvshow.idGenre = genre.idGenre) AND NOT EXISTS (SELECT 1 FROM genrelinkmusicvideo WHERE genrelinkmusicvideo.idGenre = genre.idGenre)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning country table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM country WHERE NOT EXISTS (SELECT 1 FROM countrylinkmovie WHERE countrylinkmovie.idCountry = country.idCountry)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning actor table of actors, directors and writers
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM actors WHERE NOT EXISTS (SELECT 1 FROM actorlinkmovie WHERE actorlinkmovie.idActor = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinkmovie WHERE directorlinkmovie.idDirector = actors.idActor) AND NOT EXISTS (SELECT 1 FROM writerlinkmovie WHERE writerlinkmovie.idWriter = actors.idActor) AND NOT EXISTS (SELECT 1 FROM actorlinktvshow WHERE actorlinktvshow.idActor = actors.idActor) AND NOT EXISTS (SELECT 1 FROM actorlinkepisode WHERE actorlinkepisode.idActor = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinktvshow WHERE directorlinktvshow.idDirector = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinkepisode WHERE directorlinkepisode.idDirector = actors.idActor) AND NOT EXISTS (SELECT 1 FROM writerlinkepisode WHERE writerlinkepisode.idWriter = actors.idActor) AND NOT EXISTS (SELECT 1 FROM artistlinkmusicvideo WHERE artistlinkmusicvideo.idArtist = actors.idActor) AND NOT EXISTS (SELECT 1 FROM directorlinkmusicvideo WHERE directorlinkmusicvideo.idDirector = actors.idActor)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning studio table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM studio WHERE NOT EXISTS (SELECT 1 FROM studiolinkmovie WHERE studiolinkmovie.idStudio = studio.idStudio) AND NOT EXISTS (SELECT 1 FROM studiolinkmusicvideo WHERE studiolinkmusicvideo.idStudio = studio.idStudio) AND NOT EXISTS (SELECT 1 FROM studiolinktvshow WHERE studiolinktvshow.idStudio = studio.idStudio)
09:58:31 T:1680 DEBUG: CVideoDatabase::CleanDatabase: Cleaning set table
09:58:31 T:1680 DEBUG: Mysql execute: DELETE FROM sets WHERE NOT EXISTS (SELECT 1 FROM movie WHERE movie.idSet = sets.idSet)
09:58:31 T:1680 DEBUG: Mysql commit transaction
09:58:31 T:1680 NOTICE: CVideoDatabase::CleanDatabase: Cleaning videodatabase done. Operation took 00:02
09:58:31 T:1680 DEBUG: CAnnouncementManager - Announcement: OnCleanFinished from xbmc
09:58:31 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnCleanFinished
09:58:31 T:1680 NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:04
09:58:31 T:1680 DEBUG: CAnnouncementManager - Announcement: OnScanFinished from xbmc
09:58:31 T:1680 DEBUG: GOT ANNOUNCEMENT, type: 16, from xbmc, message OnScanFinished
09:58:31 T:1680 DEBUG: Thread VideoInfoScanner 1680 terminating
09:58:31 T:6272 DEBUG: CGUIMediaWindow::GetDirectory ()

Duno if this is the problem "NOTICE: Thread VideoInfoScanner start, auto delete: false" and how to change this but "<cleanonupdate>true</cleanonupdate>" is still inside my advancedsettings.xml

Edit: Video77 isnt the problem as with XBMCSetup-20131118-0018656-master.exe it works an there was also already Video77.

Edit2: something between XBMCSetup-20131201-73a1b58-master.exe and XBMCSetup-20131201-1f67afd-master.exe went wrong and make the error as with 73a1b58 all works and with 1f67afd from the same day didnt work anymore.

What exactly is your problem?

Code:

Cleaning videodatabase done. Operation took 00:02

How is 2 seconds a long time?

Next time please pastebin the whole log. Just cutting out some lines you think might be unimportant is very confusing and might even be misleading.

Can you get us a debug log (wiki) of when this happens?

Ok, can you check this debug log bellow:

http://xbmclogs.com/show.php?id=98341

took 40+ minutes.

Thanks.

Could you try to manually trigger the clean database from Settings -> Video and see if it takes that long from there as well? There you have a progress bar which is updated for every file that is being processed.

The logic in clean database has changed to run additional checks on files that don't exist anymore. It checks if the whole source is missing and then either prompts the user about whether to remove the files from that source or to keep them or (in case of clean on update) it just assumes not to remove those files. Obviously this results in more database queries and more filesystem checks (which might be a bit slow on network shares) but unless you have a whole lot of different sources which are all unavailable at the time of the clean I can't see how it could suddenly take that long.

[tinybutstrong]

Could you try to manually trigger the clean database from Settings -> Video and see if it takes that long from there as well? There you have a progress bar which is updated for every file that is being processed.

Woot! It took like 30 seconds now. Very impressive! Log: http://xbmclogs.com/show.php?id=98419

What's the deal with clean on update in advancedsettings.xml or home clean library submenu?

[hoopsdavis]

Could you try to manually trigger the clean database from Settings -> Video and see if it takes that long from there as well? There you have a progress bar which is updated for every file that is being processed.

The logic in clean database has changed to run additional checks on files that don't exist anymore. It checks if the whole source is missing and then either prompts the user about whether to remove the files from that source or to keep them or (in case of clean on update) it just assumes not to remove those files. Obviously this results in more database queries and more filesystem checks (which might be a bit slow on network shares) but unless you have a whole lot of different sources which are all unavailable at the time of the clean I can't see how it could suddenly take that long.

Hi Montellese, should there be a setting under "Settings" -> "Video" that allows the setting of "Clean on update"? I'm running the 12/16 nightly and I'm not seeing this option in the settings.

[Montellese]

No "clean on update" still has to be enabled through advancedsettings.xml but there has always been an option under Settings -> Video -> Library called "Clean database" which can be manually triggered.

[tinybutstrong]

So, how can I get the same light speed enabling clean on update on advancedsettings.xml?

Or this is a bug?

Anyway, I can deal with this "new way" to clean the library when necessary. Problem solved for now.

Thank you for the hint!

[MrDVD]

No "clean on update" still has to be enabled through advancedsettings.xml but there has always been an option under Settings -> Video -> Library called "Clean database" which can be manually triggered.

That is the problem that non of this ways are working anymore for me for shows, since 1.12.2013.
With an xbmc from 20.11.2013 and before both way works. (Settings and advancedsettings.xml)

[hoopsdavis]

No "clean on update" still has to be enabled through advancedsettings.xml but there has always been an option under Settings -> Video -> Library called "Clean database" which can be manually triggered.

The option is no longer there in settings

[tinybutstrong]

The option is no longer there in settings

You need to enable the "Advanced" options, look screenshot bellow:

http://1.grn.cc/temp/screenshot005.png

Last menu item, Settings Level "Standard", change to Advanced.

[hoopsdavis]

The option is no longer there in settings

You need to enable the "Advanced" options, look screenshot bellow:

http://1.grn.cc/temp/screenshot005.png

Last menu item, Settings Level "Standard", change to Advanced.

Got it.

Thanks a lot