Posts: 247
Joined: Oct 2011
Reputation:
11
I am working on some additions to the JSON-API, one of the main things we need is the ability to add items, I am working on adding a few new endpoints that allow adding items, to start with movies and eventually people and streams etc.
Does anyone have an issue with this, what I mean is, it looks like adding items was left out perhaps intentionally, does anyone have an issue with adding items from scratch using the JSON-API and bypassing the scrapers all together.
Posts: 5,184
Joined: Jan 2009
Reputation:
131
Back when I first looked at adding items through JSON-RPC we didn't even have the VideoLibrary.SetFooDetails methods so simply allowing to add new items without scraper support would have left you with adding an item with only the path. That didn't make any sense so I dropped it. After that I simply never had the time to look into it again. It's probably rather simply for movies and musicvideos but could be a bit more complex for tvshows, seasons and episodes because they depend on each other. Making it work with scraper support will be quite a bit more difficult because the code that integrates our scrapers is terrifying. It usually breaks just by looking at it.
Always read the
online manual (wiki),
FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the
forum rules (wiki).
Please read the pages on
troubleshooting (wiki) and
bug reporting (wiki) before reporting issues.
Posts: 428
Joined: Oct 2008
Reputation:
18
Good to hear. Yes - we already ran into the interactions between the TV stuff when doing the directDB stuff - took us a while to sort out because if it isn't perfect it breaks in strange ways.
Posts: 5,184
Joined: Jan 2009
Reputation:
131
Yeah I've been struggling with this for a while now in my media importing work as well. But there I need to handle adding, updating and removing items and if all episodes have been removed I also need to remove the seasons that don't make any sense anymore and if all seasons have been removed I also need to remove all tvshows etc.
Always read the
online manual (wiki),
FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the
forum rules (wiki).
Please read the pages on
troubleshooting (wiki) and
bug reporting (wiki) before reporting issues.
Posts: 247
Joined: Oct 2011
Reputation:
11
Ok thanks for the confirmation. I wanted to get an idea if adding items via JSON-API and bypassing the scrappers was acceptable, it sounds like it is, I will continue to do some work on this then.
Yes I expect Shows/Seasons/Episodes will be a little more involved than movies, I was going to work on movies first and then move on to the others.
On a side note, one thing I noticed was that the request even when batch together still creates a connection to sqlite, In one of the test end points I am playing with I am passing a list of items in a single request, this means the request is a single request for say 30 items and thus the DB connection is only initialized once for that request. Do you know if this has been tried before and in your experience do you think this will help with processing speed and overall system latency (gui responsiveness).
Posts: 5,184
Joined: Jan 2009
Reputation:
131
I've always found it a bit odd that we are opening and closing database connections all the time all over the Kodi core code. But I have no idea if connection-pooling would really improve performance / responsiveness.
Furthermore in case of JSON-RPC requests this would be quite tricky as requests need to be independent from each other (even when batched). Furthermore at the place where we know that we are handling a batch request we don't really know which requests require which database connection (if any at all) and you can't assume that if the first request requires a db connection so will any/all the other requests. And at the place where we handle individual requests we have absolutely no clue that there are any other requests so you don't know that you should leave the db connection open and even worse you don't know when to close it.
So you'd probably have to write very specific code into a place which is designed to be as generic as possible i.e. deserialize a JSON-RPC request, lookup the implementation, execute it, serialize the JSON-RPC response (without any knowledge about the actual method implementation).
Always read the
online manual (wiki),
FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the
forum rules (wiki).
Please read the pages on
troubleshooting (wiki) and
bug reporting (wiki) before reporting issues.
Posts: 247
Joined: Oct 2011
Reputation:
11
I guess all I was trying to say was that in some requests like SetMovieDetails you could pass in an array of movies that are going to be updated, it could open the connection do all the work and close the connection. This is better than batched requests where each one would need to open set the movie and then close, then open again for the next request etc.
Anyway I will have a play and see if it makes any difference.
Posts: 3,077
Joined: Jun 2009
(2015-09-15, 10:08)Montellese Wrote: I've always found it a bit odd that we are opening and closing database connections all the time all over the Kodi core code. But I have no idea if connection-pooling would really improve performance / responsiveness.
This is the cause of some problems with Mysql performance, and major concern for Rpi users
You can queue an album in seconds since it's one open / close but it takes ages to queue all files 1 by 1 since it opens / close query items each times and opening a big db on slow sdcard is well very slow.
There was long discussion last year about that but nobody never took a final decision about the need.
Posts: 296
Joined: Nov 2014
Reputation:
10
2015-09-16, 04:08
(This post was last modified: 2015-09-16, 04:25 by angelblue05.)
The way it's done in the add-on currently is one opened database connection to process multiple items and it's blazing. I am positive the way null_pointer is doing things will improve the performance overall. I've seen opening and closing database connections at large scale and it really slows things down (which is where we hit a wall with json-rpc the first time around).
Posts: 3,077
Joined: Jun 2009
Well writing simultaneously will always be a problem, but from an sqlite point of point of view one connection and WAL mode will always be much more efficient than normal journal mode and opening closing.
WAL mode with proper transaction would also make it way easier for you to detect locks and delay one of the sync.
I have no knowledge to show that in Kodi, but in Yatse the gains were quite big.
Posts: 5,184
Joined: Jan 2009
Reputation:
131
In my media importing there are no simultaneous writes. For every imported media type (movies, tvshows, seasons, episodes, musicvideos) I first read all the previously imported items from the db, compare them to the received/imported items (doesn't have anything to do with the database) and then start a db transaction and update one item at a time in that transaction. And different media types are handled sequentially (because you need to import tvshows before seasons before episodes or you'll get a huge mess). So it definitely is also a problem for simultaneous reads and writes.
Concerning WAL mode IIRC it does not work over network filesystems and with Kodi it is possible to store your userdata on a remote server and set up path substitution.
Always read the
online manual (wiki),
FAQ (wiki) and search the forum before posting.
Do not e-mail Team Kodi members directly asking for support. Read/follow the
forum rules (wiki).
Please read the pages on
troubleshooting (wiki) and
bug reporting (wiki) before reporting issues.