[akashb]

Hello folks!

First of all, this is my first post, so please be gentle. If this is in the wrong section of the forum, please let me know how to move it or have a moderator do it.

I run XBMCbuntu on an i5 Intel NUC as my HTPC and have been immensely pleased so far. I would like to thank the developers behind this amazing piece of software.

PROBLEM:
---------------
I was having some issues with my library updates slowing down over a period of time, mostly the 'Clean Library' feature. My library has ~200 movies and ~400 TV show episodes with more being slowly added via automated downloads. I have the XBMC Library auto update plugin which automatically cleans and updates the library every 24 hours but then I started noticing that the clean library step takes increasingly longer with each passing day.

To attempt to fix this, I would re-create my library (by deleteting MyVideos.db and scanning again), and the clean library step would be fixed taking 3-5 seconds to complete. But as days pass, it would again slow down and take minutes instead of seconds.

I recently configured my XBMC installation to use a MySQL database in the hopes of resolving this issue but it persisted. Today, I decided to spend some time and find out the reason behind this behavior.

ISSUE:
---------

I discovered that in the 'files' table of the MyVideos database, there are entries that have the strFileName field set to URLs from plugins. There are hundreds of such entries which correspond to my usage of different video plugins. Below are some examples:
plugin://plugin.video.tvondesizonexl/?videoLink=http%3A%2F%2Fwww.videoweed.es%2Ffile%2F2085c067a210b
plugin://plugin.video.ign_com/?url=http%3A%2F%2Fwww.ign.com%2Fvideos%2F2014%2F10%2F02%2Fmiddle-earth-shadow-of-mordor-6-tips-for-mastering-the-nemesis-system&mode=playVideo

I manually removed a few hundred of these entries and immediately noticed that the clean library step started speeding up again as I kept on removing these entries. I am fairly confident that this is causing the issue of slowing down the clean library step, at least in my case. I suspect that the XBMC logic finds these URLs and attempts to verify those files exist or something like that, thereby spending time during the clean library step.

These are plugin URLs and not files per se, so why are they being stored in the database?

Hope this is helpful and a developer can figure this out.

Thank you.
Akash

[nickr]

I think they are probably stored there to provide watched status etc.

[un1versal]

No need to manually remove anything, iirc texturecache.py is fantastic for clearing up all-sorts of crud off databases. I cant remember what the options passed to the texturecache.py are for this particular thing perhaps try asking in the texturecache thread.. or run it without any options and you get a list of functions it supports.

[akashb]

Thanks for the responses guys!

@nickr, I think it makes sense. I do see videos inside some plugins being marked as seen.

@uNiversal, I already have texturecache.py but could not find an option to clear stuff in the files table. I think it only affects other tables that are relevant to cached thumbnails, artwork and other images. The problem I think is in the core table that contains file lists, not cache entries. Please let me know if what I'm saying is not accurate.
Also, when I say I manually removed stuff, I meant I did that by running SQL queries on the database (which is what texturecache.py would do but in a more efficient and hassle-free way) and not one entry at a time.

[un1versal]

Im not entirely sure best person would be developer and maybe he can add that option I think it can export/import watched already so maybe it can do everything else, if not maybe you wanna ask the dev if he is willing to add it. It goes well deeply beyond art cleaning now that's the impression I get. He has other tools too maybe these can do it?

[akashb]

Thanks for the response uNiversal.
I'll post on the thread and see if the dev is interested in implementing something like this.

However, I believe that this should not happen in XBMC in the first place. Its okay to store the watch status for plugin URLs in the XBMC database, but not okay to have those URLs in the files table (containing file names of stuff on disk) which impacts performance of library clean/update functions. Since most plugins update content dynamically every time you open them, that content should be treated separately from the persistent content on disk in your library. Let me know if you think this is not a good idea.

[un1versal]

Garbage collection is not the best, one can only get it to some people and see if gets fixed, or if anyone has skills fix it themselves.

[menakite]

If you check your xbmc.log you'll see lots of

Code:

21:57:44 12634.439453 T:3038126080 WARNING: CreateLoader - unsupported protocol(plugin) in ...

A library clean checks if the file/URL still exists, but there's no handler for plugin:// URLs [1] so they're kept there. It's a known issue.

[1] https://github.com/xbmc/xbmc/blob/master...#L120-L212

[akashb]

Thank you uNiversal for a response and menakite for pointing out the root of the problem (I tried to find an option to give you rep but couldn't figure out where it is).

menakite, now that I understand the issue, I might try to play with the source and see if I can come up with a workaround when time permits. But would you happen to know why these entries are put in the database in the first place?

Thanks!
Akash

[menakite]

I'm not sure. I think it's to provide watch status and a resume point. I didn't investigate further.

I'm not even sure what the correct approach would be with "dynamic" content - asking the plugin for each entry is of course not really possible. It seems sane to, at least, delete entries from plugins that have been uninstalled though.

[akashb]

Thanks menakite.

For now, I wrote a simple shell script that removes any entry with a '?' character or with a '.flv' extension in the strFileName column from the 'files' table in the database. It runs through a cron job every 24 hours and seems to work great for my needs.

---

Now that I think about it, we could maybe have an extra column in this table which acts as a flag for dynamic content? If this flag is set, XBMC should not try to verify the existence of the file during clean/update. And this flag is set for any entry that is pushed into the database through a plugin. This could also optimize clean by removing any rows which have the flag set and are older than a set time threshold (like 60 days).

Just some random ideas, obviously devs may have a better explanation of why this approach is not acceptable.