[kricker]

I will attempt to recreate Nfusion2011's issues tomorrow and report my findings.

[sjw1]

With an attitude like nfusion's, you guys have the patience and manners of Saints...

[eddiem]

With an attitude like nfusion's, you guys have the patience and manners of Saints...

I'll second that, you would think he had paid for support!!

[gensufz]

I don't have an attitude, but I get annoyed, when I'm told to do the same thing that I've already done 3-4x (with pictures if I may add).

I'm not sure why you are unable to reproduce it, I'm using the 10.1 version if that helps...

Its a pretty straight forward bug:

When movies are "CLEANED" from the database after the source is removed, it is implied that your database is 100% empty, yet a whole bunch of junk data remains in the .db file as shown by the screenshots and the evidently large file, and the fact that it still takes 2minutes to clean a database that should be 100% empty!

Over time, this grows and grows and grows and grows. This ends up with a relatively big file that takes MINUTES to do a clean, even if the database is 100% empty because it contains 8MB of junk data.

This conversation also seems to be going 1 way, I asked you twice jmarashall why the database would be "designed" to keep junk data (file paths) of movies that no longer exist, to which I received no reply. Its a lot easier to blame a problem on "design" than acknowledge its there and fix it.

P.S. No, I know i didn't pay for support, but if people don't put out the deficiencies of the program, how are the developpers expected to fix it? I think I've done more than my due diligence in pointing it out, and even providing pictures as proof... I think asking for my entire collection archive contents is a bit unreasonable.

[kricker]

I understand your frustration. Give me tonight to reproduce the issues. If I can reproduce it I'll create the necessary Track ticket with accompanying logs. Everyone just take a second to take a breath. I believe you documented the steps clear enough. If I have issues I'll PM you for more info.

[bradvido88]

When movies are "CLEANED" from the database after the source is removed, it is implied that your database is 100% empty, yet a whole bunch of junk data remains in the .db file as shown by the screenshots and the evidently large file, and the fact that it still takes 2minutes to clean a database that should be 100% empty!

The database is not used exclusively for videos in your library, so cleaning it with an empty library should not result in an "100% empty" database. It doesn't clean out the entries from the path or file tables because you can still access those videos in "file view" in XBMC, and XBMC still tracks resume points and watched flags for videos in files view (a feature that most users like).

[kricker]

So from reading over the thread multiple times, it appears there are 2 main issues being discussed. The video database and the thumbnails folder.

Issue 1 the Thumbnail folder.
Orphaned items in the thumbnail folder has been a known issue for some time and anyone is welcome to suggest how to better clean this folder. Thanks to Bradvido88 there is now a script to help with this. The hope is this will eventually be done from within XBMC itself preferably with the "clean library" function or when a source is removed.

Issue 2 the video Database file.
The database itself is something that has been hacked and tweaked over the years. Even now it is undergoing a re-work for various improvements.

Part of Nfusion2011's issue is that the database doesn't get cleaned sufficiently when sources/files are removed. This is the item I have just tested. I tested using a nightly build from March 14th Git:0897223, as that is what I currently have loaded on my dev PC. I can retest with a vanilla 10.1 install if need be. I started with a fresh new empty video database file and no video sources of any kind. Here was my process:

View database file and confirmed it is empty of any data
Added a video source in XBMC and scraped the info for that source. (CSI-Crime Scene Investigation, Season 11 containing 18 episodes)
exit XBMC
Viewed database file to view the new data
run XBMC / remove source from XBMC and let it clean the library on removal (now there are 0 video sources)
exit XBMC
viewed database to see what data remains (list below)
run XBMC / settings/video/Library/ Clean Library
exit XBMC
viewed database to see what data now remains (list below)

Here are the results:

Source added and scraped.

Video database contains items in the following tables:

actors 31 items
path 2 items
files 18 items
tvshow 1 item
actorlinktvshow 13 items
studiotvlinkshow 1 item
episode 18 items
tvshowlinkepisode 18 items
tvshowlinkpath 1 item
actorlinkepisode 11 items
directorlinkepisode 4 items
writerlinkepisode 5 items
genrelinktvshow 1 item
studio 1 item
streamdetails 37 items
genre 1 item

Texture database contains items in the following tables:

texture 86 items

Source removed and library auto cleaned upon removal.

Video database still contains items in the following tables:

actors 31 items
path 3 items
files 18 items
writerlinkepisode 5 items
studio 1 item
genre 1 item

Texture database contains items in the following tables:

texture 86 items

Video Library clean command run from settings.

Video database still contains items in the following tables:

actors 5 items
path 1 item
files 18 items
writerlinkepisode 5 items

Texture database contains items in the following tables:

texture 86 items

I have copies of the database as well as the thumbnails folder and textures database at each step; new, source added and scraped, source removed with auto clean, and clean library command run from settings. I can share these with anyone who would like to see them.

[jmarshall]

Post a ticket on trac.

[kricker]

Will do in the AM after I've gotten some sleep I'd make a mess out of it tonight I am sure.

[ashlar]

Wow, kricker, awesome investigation! Thanks a lot!

[kricker]

Ticket filed.

[gensufz]

So from reading over the thread multiple times, it appears there are 2 main issues being discussed. The video database and the thumbnails folder.

Issue 1 the Thumbnail folder.
Orphaned items in the thumbnail folder has been a known issue for some time and anyone is welcome to suggest how to better clean this folder. Thanks to Bradvido88 there is now a script to help with this. The hope is this will eventually be done from within XBMC itself preferably with the "clean library" function or when a source is removed.

Brad's script comes nowhere close to cleaning the junk data, it does do a much better job that what XBMC does, but it still has a long way to go to clean. In my case for instance, my 1GB Roaming folder only went down to about 850MB. It should have gone down to about 190-200MB.

What XBMC needs to do is keep a record of which jpg files are actually being USED for widescreen banners, fanarts, and posters, and season thumbs. Delete the rest, I'm not sure why this hasn't been done or needs feedback from the community. I'm not a programmer, but the logic seems pretty straight forward: "If its not in use, its junk, remove it, the program will re download it when its needed."

Issue 2 the video Database file.
The database itself is something that has been hacked and tweaked over the years. Even now it is undergoing a re-work for various improvements.

Good to know!

Part of Nfusion2011's issue is that the database doesn't get cleaned sufficiently when sources/files are removed. This is the item I have just tested. I tested using a nightly build from March 14th Git:0897223, as that is what I currently have loaded on my dev PC. I can retest with a vanilla 10.1 install if need be. I started with a fresh new empty video database file and no video sources of any kind. Here was my process:

View database file and confirmed it is empty of any data
Added a video source in XBMC and scraped the info for that source. (CSI-Crime Scene Investigation, Season 11 containing 18 episodes)
exit XBMC
Viewed database file to view the new data
run XBMC / remove source from XBMC and let it clean the library on removal (now there are 0 video sources)
exit XBMC
viewed database to see what data remains (list below)
run XBMC / settings/video/Library/ Clean Library
exit XBMC
viewed database to see what data now remains (list below)

Here are the results:

Source added and scraped.

Video database contains items in the following tables:

actors 31 items
path 2 items
files 18 items
tvshow 1 item
actorlinktvshow 13 items
studiotvlinkshow 1 item
episode 18 items
tvshowlinkepisode 18 items
tvshowlinkpath 1 item
actorlinkepisode 11 items
directorlinkepisode 4 items
writerlinkepisode 5 items
genrelinktvshow 1 item
studio 1 item
streamdetails 37 items
genre 1 item

Texture database contains items in the following tables:

texture 86 items

Source removed and library auto cleaned upon removal.

Video database still contains items in the following tables:

actors 31 items
path 3 items
files 18 items
writerlinkepisode 5 items
studio 1 item
genre 1 item

Texture database contains items in the following tables:

texture 86 items

Video Library clean command run from settings.

Video database still contains items in the following tables:

actors 5 items
path 1 item
files 18 items
writerlinkepisode 5 items

Texture database contains items in the following tables:

texture 86 items

I have copies of the database as well as the thumbnails folder and textures database at each step; new, source added and scraped, source removed with auto clean, and clean library command run from settings. I can share these with anyone who would like to see them.

That's perfect, however I would just like to add that the problem obviously is not specific to just TV Shows, and affects movies as well. Some other elements that are not TV specific could, and most likely would be left over when the process above is followed.

I guess my major peeve in all this is the time it takes to CLEAN. When XBMC is first installed, cleaning takes less than a second. As I add more and more and more and more, logically, cleaning takes longer.

It stands to reason that as I remove and remove and remove and remove, cleaning should go faster, this doesn't seem to be the case. The time it takes to clean, only increases. I personally have never seen the myvideos.db file ever decrease in filesize, even though it contains 0 sources, and 0 videos. This is what causes the problem I think.

[gensufz]

P.S.

Thank you kricker for taking the time to confirm my findings.

[bradvido88]

Brad's script comes nowhere close to cleaning the junk data, it does do a much better job that what XBMC does, but it still has a long way to go to clean.

I've updated the script to clean out a lot more junk. Link is in my signature.

[kricker]

The database came as a result of many user requests. No one on the team was a database expert, which I believe is still the case. The developers have worked and worked on it. Yes there are issues they are aware of and some they may not be. JMarshall has taken the reigns of the overhaul. I'd be happy to help those who need to document issues they are having in order to create Trac tickets.

There have been threads on cleaning the thumbnails folder in the past. This isn't the first time it has been discussed. The main problem before, which probably still applies, is that the db wasn't really good at keeping track of what image was or was not in use. At least I think it was something like that.

It is east to say how it "should" work. It is another matter entirely programming it to work as such. If there is a database expert around who can lend a hand it would be helpful.

Yes I am sure the lack of cleaning applies to movies. I did't test movies as I figured the cleaning process would be the same. It may even be the same for music.