[BePov]

Hi,

I'm trying to create my first addon, so my XBMC-vocabulary may not be precise yet, feel free to correct it .

I'm trying to create an addon that will search through my "source" directories, and display the files from those directories. I also want to display the metadata for files that were already added to my database (scrapped). The behavior I'm looking for is quite similar to what can be found in the "Videos/files" menu in XBMC. See image below.



I'm mostly interested in the informations displayed on the left (when available) and on the "Movie Information" tab in the context menu. My goal is to apply for each file a specific process, and then to hide the processed files (like the watched/unwatched status). Maybe there is a way to include that directly in the file browser, but I'm not aware of that.

Now, let's start with what I already tried :

I first tried to reproduce that behavior for files that were already added to my database (retrieved through the jsonrpc method "VideoLibrary.GetMovies").
Once i retrieved my list of movies, I displayed them using :

Code:

item = [one line of the result from a "VideoLibrary.GetMovies" call]
url = item['file"]
filename = ntpath.basename(url)
li = xbmcgui.ListItem(filename, thumbnailImage=item["thumbnail"])
li.setInfo(type='video', infoLabels={'year':item["year"],
                             'title': item["label"],
                             'director': str(item["director"]),
                             'plot': item["plot"] })
                            
xbmcplugin.addDirectoryItem(handle=addon_handle, url=url, listitem=li)

However, when doing so, I'm not getting the "Movie information" tab in the context menu. I still have the other menu items though ("mark as watched", "add to favourites", "play", etc...), and i can play the file from there, so I guess the link is correct.
What am I missing ? Is it possible to achieve ?

Then, I list files retrieved from a call to the jsonrpc method "Files.GetDirerctory" (for a given source). I'm then trying to retrieve metadata for these files from the previous list.
But this is very long, since I have to manually find which metadata set corresponds to each file (which means lots of lookups in a large dict - very slow, especially in python)

Is there a more straightforward way to reproduce the behavior of XBMC's native file browser ? Maybe something to directly retrieve metadata associated to a given file if any ?

Thanks in advance,
BePov

[ironic_monkey]

you have to set the content to movies. xbmcplugin.setContent(handle, 'movies')

[BePov]

Hi,

thanks for the reply.

I already have this at the begining of my script (copied from tutorials) :

Code:

base_url = sys.argv[0]
addon_handle = int(sys.argv[1])
xbmcplugin.setContent(addon_handle, 'movies')
args = urlparse.parse_qs(sys.argv[2][1:])

And then, I'm using the same "addon_handle" for the rest of my script. My context menu contains
- Queue item
- Play from here
- Add to favourites
- Mark as watched
- Add-on settings

whether I set content or not

Also, my addon is a video addon

[ironic_monkey]

that sounds all correct. something must have regressed. no computer just my phone but will see if i find some time to test soonish.

[BePov]

Ok, thanks

I just noticed that even though the "play from here" works, the "Mark as watched" menu item actually doesn't do anything. It feels like I'm not giving the correct data to the ListItem. Actually, I think I'm missing something to establish a link between what I'm displaying and the movie from the database...

In the mean time, I've been benchmarking my script to find bottlenecks.

It seems the data retrieval from the database was the bottleneck :
I was requesting many data in my call to VideoLibrary.GetMovies, and it seems that grabbing some of them takes a LOT of time.

Just for information , here's a list of the times it took to retrieve specific data with the following code (only the "properties" changes ) :

Code:

    print "base"
    t = time.time()
    jsonResult = xbmc.executeJSONRPC('{"jsonrpc": "2.0", "method": "VideoLibrary.GetMovies",  "id": 1}')
    print time.time() - t
    
    print "art"
    t = time.time()
    jsonResult = xbmc.executeJSONRPC('{"jsonrpc": "2.0", "method": "VideoLibrary.GetMovies", "params": {"properties": ["art"]}, "id": 1}')
    print time.time() - t
        ....

And the results are quite long (times are in seconds) :

Code:

19:05:55 T:1236  NOTICE: base
19:05:57 T:1236  NOTICE: 2.01799988747
19:05:57 T:1236  NOTICE: art
19:06:10 T:1236  NOTICE: 12.8029999733
19:06:10 T:1236  NOTICE: thumbnail
19:06:19 T:1236  NOTICE: 9.22200012207
19:06:19 T:1236  NOTICE: file
19:06:21 T:1236  NOTICE: 1.90799999237
19:06:21 T:1236  NOTICE: plot
19:06:24 T:1236  NOTICE: 2.16000008583
19:06:24 T:1236  NOTICE: cast
19:07:25 T:1236  NOTICE: 61.4559998512
19:07:25 T:1236  NOTICE: director
19:07:27 T:1236  NOTICE: 2.15199995041
19:07:27 T:1236  NOTICE: year
19:07:29 T:1236  NOTICE: 1.96099996567
19:07:29 T:1236  NOTICE: all
19:09:01 T:1236  NOTICE: 91.4510002136
19:09:01 T:1236  NOTICE: "file", "plot", "director", "year"
19:09:03 T:1236  NOTICE: 2.22600007057

Is it normal ? I'm mostly surprised by the times it takes for art/thumbnail, which are longer to retrieve than plot for example, even though there is less text to get (Unless I'm mistaken, they only contain paths to images).

[Lunatixz]

Ok, thanks

I just noticed that even though the "play from here" works, the "Mark as watched" menu item actually doesn't do anything. It feels like I'm not giving the correct data to the ListItem. Actually, I think I'm missing something to establish a link between what I'm displaying and the movie from the database...

In the mean time, I've been benchmarking my script to find bottlenecks.

It seems the data retrieval from the database was the bottleneck :
I was requesting many data in my call to VideoLibrary.GetMovies, and it seems that grabbing some of them takes a LOT of time.

Just for information , here's a list of the times it took to retrieve specific data with the following code (only the "properties" changes ) :

Code:

    print "base"
    t = time.time()
    jsonResult = xbmc.executeJSONRPC('{"jsonrpc": "2.0", "method": "VideoLibrary.GetMovies",  "id": 1}')
    print time.time() - t
    
    print "art"
    t = time.time()
    jsonResult = xbmc.executeJSONRPC('{"jsonrpc": "2.0", "method": "VideoLibrary.GetMovies", "params": {"properties": ["art"]}, "id": 1}')
    print time.time() - t
        ....

And the results are quite long (times are in seconds) :

Code:

19:05:55 T:1236  NOTICE: base
19:05:57 T:1236  NOTICE: 2.01799988747
19:05:57 T:1236  NOTICE: art
19:06:10 T:1236  NOTICE: 12.8029999733
19:06:10 T:1236  NOTICE: thumbnail
19:06:19 T:1236  NOTICE: 9.22200012207
19:06:19 T:1236  NOTICE: file
19:06:21 T:1236  NOTICE: 1.90799999237
19:06:21 T:1236  NOTICE: plot
19:06:24 T:1236  NOTICE: 2.16000008583
19:06:24 T:1236  NOTICE: cast
19:07:25 T:1236  NOTICE: 61.4559998512
19:07:25 T:1236  NOTICE: director
19:07:27 T:1236  NOTICE: 2.15199995041
19:07:27 T:1236  NOTICE: year
19:07:29 T:1236  NOTICE: 1.96099996567
19:07:29 T:1236  NOTICE: all
19:09:01 T:1236  NOTICE: 91.4510002136
19:09:01 T:1236  NOTICE: "file", "plot", "director", "year"
19:09:03 T:1236  NOTICE: 2.22600007057

Is it normal ? I'm mostly surprised by the times it takes for art/thumbnail, which are longer to retrieve than plot for example, even though there is less text to get (Unless I'm mistaken, they only contain paths to images).

No you are correct, Json requests for art takes a while, you should consider caching the returns... seems silly but it is faster to retrieve the cached data then to request new...

[BePov]

Very good point there ! I'll do that. I might actually even set up a service that would update that cache on a regular basis or on user input. Is there anything in XBMC designed to store cached data for movies/tv shows ?


More importantly, I'm still struggling with the "movie information" stuff :

I added a MenuContext Item that would call ActivateWindow("movieinformation"), but that will only display an unrefreshed window : it will contain either an empty window or the last movie information I've displayed from the actual movie list.

Side note : When pressing 'i' in my addon, the "movie information" window also appears, however, it only contains the data i've inserted in the ListItem, and the "Refresh" button is greyed out and I can't update the movie from here (which is the point of my addon).

I've seen this issue on multiple threads here, but unfortunately, none of them had any answers. I'd be really grateful if someone had clues on how to achieve this, or even examples of addons doing the same kind of things.

Thanks
BePov

[RockerC]

seems silly but it is faster to retrieve the cached data then to request new.

Sarcasm I hope? I mean acceleration is the whole point with a cache and caching.

[Lunatixz]

seems silly but it is faster to retrieve the cached data then to request new.

Sarcasm I hope? I mean acceleration is the whole point with a cache and caching.

Not sure a disk based cache should be faster then a json query... at minimum they should be on par. After all json has fresh info, cache is stale... So caching json is kinda silly but for now it works.

[BePov]

Not sure a disk based cache should be faster then a json query... at minimum they should be on par. After all json has fresh info, cache is stale... So caching json is kinda silly but for now it works.

For queries that big, it really makes sense : They last more than 1 minute on my db, while the fetching from disk only takes 2 seconds.

In the meantime, I've also advanced a bit on my project, here are a few more notes ;D...

Currently, checking your whole database is quite a tedious task, and most of the time, external tools are the best/easier solutions.
The goal of my addon is to try and tackle that : It displays a list of every movie / TV Show from the disk, and allows user to check if they were correctly found by the scrappers. Once user has checked a file, it won't appear anymore in the listing, which will allow to check your whole database bits by bits (if you have a big database, this can be useful to resume the process later). This also allows to easily check the newly added files in your db later on.

User can check in a single list which data were retrieved for each movie/tvshow. If no data was retrieved, or if the scrapper didn't find the correct information, user should be able to change it (just like in the file browser - this is the part I'm having trouble with). Then, when user is satisfied with the data associated to a movie/tvshow, it can mark the file as checked, and the file will not appear anymore in the list of files to be checked.

So far, everything seems to work nicely, and only the file browsing part is still troublesome.

However, I'm not very fond of some hacks I had to use to perform this :

1 - Since i couldn't find how to access and modify metadata cache, I have created a service that handles the list of movies / files / sources / checked files. This was the best solution I found to be able to load and display that much data in a short time. This service first loads what it had last time it was started (to have some data to display as quickly as possible), and then retrieves data from the db to update itself.
2 - I've found a few service addons that could actually do some cache management, but I couldn't figure out how to communicate with them from my addon (especially stuff like xbmc common cache which would have been very interesting https://github.com/HenrikDK/xbmc-common-cache ). How to store / retrieve stuff from their StorageServer ?
3 - Since I couldn't use them right now, I've used Window.setProperty to communicate between my addon and my service. The addon sets properties on the main window, and the service has a loop checking states of those properties. (inspired by LazyTv). I had hopes that maybe I could share an object instance or even strings between my addon and my service instead of using an unrelated window.
4 - Even though my service knows the whole database, I've only displayed a given number of items at a time because creating the list for the entire database is very long.

Points (1) and (4) are very surprising for me, because XBMC is already able to display huge lists (very) quickly with a lot of information. I'm sure there is something wrong with the way I'm displaying movies / TvShows.

Feel free to provide comments on the project (maybe something similar already exists, or it's just useless for most people, and I'm just doing stupid things ), or suggestions, or even help on the above points. Any of those would be appreciated !

[Hedda]

2 - I've found a few service addons that could actually do some cache management, but I couldn't figure out how to communicate with them from my addon (especially stuff like xbmc common cache which would have been very interesting https://github.com/HenrikDK/xbmc-common-cache ). How to store / retrieve stuff from their StorageServer ?

Ah, maybe post here http://forum.kodi.tv/showthread.php?tid=116496

And maybe related threads
http://forum.kodi.tv/showthread.php?tid=172531
http://forum.kodi.tv/showthread.php?tid=185513


Also, checkout this pull request #6202 for a new webserver image resizing feature: https://github.com/xbmc/xbmc/pull/6202

Could be a little off-topic here but might be good to know for future reference.

[BePov]

I don't know how I managed to miss their page on the wiki AND on the forum... I must be tired
http://kodi.wiki/view/Add-on:Common_plugin_cache contains everything I need to use it

Thanks for that ! I'll try it as soon as possible !