Login at Kodi Home

cg110 · 2014-02-24, 03:18

There won't be in your queries. The issue with cast is related to other addons and what they request. Eg if you included cast, streamdetails, tag or showlink, you'd find that a query for the cast would fire. In fact any of them triggers the inclusion of the others.

I guess what it comes down to is my expectation of the queries thrown at JSON should turn into the minimum amount of work to get that result. At times they're not, I suspect it's because most APIs want the full objects.

As for individual PRs I suspect some parts will break up ok, but some bits will have dependencies on each other.

I also don't know what the scope is for some of these changes at this stage of development. 13.0 is heading for beta, so I guess smaller scoped items may stand a better chance. Of course that means more fun with git, and how to get branches with just the right set of changes in...

Milhouse · (This post was last modified: 2014-02-24, 09:35 by Milhouse.)

Ah yes, if I change the JSON properties to "title, art, file, tag", I see the following query being executed - that's not good (ideal):

Code:

01:41:11 29912.289062 T:2844783696    INFO: JSONRPC Server: New connection added

01:41:11 29912.291016 T:2844783696   DEBUG: JSONRPC: Incoming request: {"jsonrpc": "2.0", "params": {"sort": {"order": "ascending", "method": "title"}, "filter": {"operator": "contains", "field": "title", "value": "azkaban"}, "properties": ["title", "art", "file", "tag"], "limits": {"start": 0, "end": 35}}, "method": "VideoLibrary.GetMovies", "id": "libMovies"}

01:41:11 29912.464844 T:2844783696    INFO: JSONRPC Server: Disconnection detected



140224  1:41:11 93037 Connect   [email protected] on

                93037 Query     SET NAMES utf8

                93037 Query     show databases like 'MyVideos78'

                93037 Query     show tables

                93037 Init DB   MyVideos78

                93037 Query     show databases like 'MyVideos78'

                93037 Query     show tables

                93037 Query     select * from movieview  WHERE ((movieview.c00 LIKE '%azkaban%'))

                93037 Query     select * from movieview where idMovie=449

                93037 Query     SELECT actors.strActor,  actorlinkmovie.strRole,  actorlinkmovie.iOrder,  actors.strThumb,  art.url FROM actorlinkmovie  JOIN actors ON    actorlinkmovie.idActor=actors.idActor  LEFT JOIN art ON    art.media_id=actors.idActor AND art.media_type='actor' AND art.type='thumb' WHERE actorlinkmovie.idMovie=449 ORDER BY actorlinkmovie.iOrder

                93037 Query     SELECT tag.strTag FROM tag, taglinks WHERE taglinks.idMedia = 449 AND taglinks.media_type = 'movie' AND taglinks.idTag = tag.idTag ORDER BY tag.idTag

                93037 Query     select * from movielinktvshow where idMovie=449

                93037 Query     SELECT * FROM streamdetails WHERE idFile = 449

                93038 Connect   [email protected] on

                93038 Query     SET NAMES utf8

                93038 Query     show databases like 'MyVideos78'

                93038 Query     show tables

                93038 Init DB   MyVideos78

                93038 Query     show databases like 'MyVideos78'

                93038 Query     show tables

                93038 Query     SELECT type,url FROM art WHERE media_id=449 AND media_type='movie'

                93038 Quit

                93037 Quit

It seems that cast, tag and streamdetails are always retrieved whenever one or more of the properties is specified.

It also appears that the cast/tag/streamdeatils query and the art query ("select type, url from art") take place using separate connections.

Milhouse · 2014-02-24, 03:34

(2014-02-24, 03:18)cg110 Wrote: As for individual PRs I suspect some parts will break up ok, but some bits will have dependencies on each other.

I also don't know what the scope is for some of these changes at this stage of development. 13.0 is heading for beta, so I guess smaller scoped items may stand a better chance. Of course that means more fun with git, and how to get branches with just the right set of changes in...

I'd suggest treating the update optimisation, and any cast retrieval optimisation as two separate PRs. I have trouble with git myself so am in no position to advise but others here are usually quite helpful when it comes to creating PRs and squashing commits and whatnot.

**Montellese** · 2014-02-24, 08:52

Yes you are right, it simply grabs all the extra information if one of the extra properties is requested. That's because that's how it works internally and nobody wanted to dig into it any deeper.

I'd also recommend splitting your work into separate PRs even if some things depend on others. Otherwise it will be a lot more difficult to get your work into mainline. If someone doesn't like a part of your implementation it will block your whole work which you can avoid by doing separate PRs. If you have any git questions feel free to send me a message.

cg110 · 2014-03-01, 17:24

FYI, I've now put the code changes for finer grained updates into a topic branch:
https://github.com/cg110/xbmc/commits/vi...ed-updates

I've built them on windows and linux, and they build cleanly. I want to just do a bit of testing, then I'll make the pull request for ticket:
http://trac.xbmc.org/ticket/14967

Milhouse · 2014-03-01, 20:49

I've built OpenELEC on R-Pi with PR4301, and it seems to be behaving as before (ie. very well!)

Example: Updating 320 movies with refreshed imdb rating and votes - without optimisation: 36 minutes, with optimisation: 25 seconds... Smile

Do you plan on adding a second PR for the cast query optimisation?

Milhouse · (This post was last modified: 2014-03-01, 21:49 by Milhouse.)

One word of caution when using PR4301 - if applying a mass of updates (ie. imdb rating and votes) via JSON while the user is sat in the movie library view, the Pi will run out of memory and appear to hang. I've got swap enabled, otherwise XBMC would probably crash due to OOM, however it's likely that if I let this run for long enough it would use all my swap too (it's already using 88% of 128MB swap on a 512MB Pi, with 256MB physical RAM and 256MB for GPU).

If the user isn't in the library view (ie. sat at the home menu), then there is no problem when applying the same updates - hardly any memory is used at all.

In my case when applying 320 updates remotely from a PC with the Pi as the client and the user viewing the Movies library view (thumbnails, stock Confluence), the Pi will stop responding after about item 215, with all physical memory exhausted and swap memory increasingly used.

Milhouse · 2014-03-01, 22:05

Yep, eventually xbmc got the chop from the oom-killer...

Code:
[ 5507.341883] systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

[ 5507.341915] CPU: 0 PID: 1 Comm: systemd Tainted: G        W    3.13.5 #1

[ 5507.341924] Backtrace:

[ 5507.341966] [<c00113d0>] (dump_backtrace+0x0/0x10c) from [<c00116a8>] (show_stack+0x18/0x1c)

[ 5507.341975]  r6:000201da r5:cf838008 r4:00000000 r3:00000000

[ 5507.342001] [<c0011690>] (show_stack+0x0/0x1c) from [<c06256dc>] (dump_stack+0x20/0x28)

[ 5507.342028] [<c06256bc>] (dump_stack+0x0/0x28) from [<c061f7fc>] (dump_header.isra.17+0x94/0x1c4)

[ 5507.342062] [<c061f768>] (dump_header.isra.17+0x0/0x1c4) from [<c007465c>] (oom_kill_process+0x2b4/0x444)

[ 5507.342081] [<c00743a8>] (oom_kill_process+0x0/0x444) from [<c0074cc8>] (out_of_memory+0x298/0x2e0)

[ 5507.342099] [<c0074a30>] (out_of_memory+0x0/0x2e0) from [<c0078a04>] (__alloc_pages_nodemask+0x818/0x854)

[ 5507.342116] [<c00781ec>] (__alloc_pages_nodemask+0x0/0x854) from [<c0072c24>] (filemap_fault+0x24c/0x484)

[ 5507.342137] [<c00729d8>] (filemap_fault+0x0/0x484) from [<c008e6f0>] (__do_fault+0x70/0x4ac)

[ 5507.342154] [<c008e680>] (__do_fault+0x0/0x4ac) from [<c00909c0>] (__handle_mm_fault+0x15c/0x820)

[ 5507.342174] [<c0090864>] (__handle_mm_fault+0x0/0x820) from [<c0092a60>] (handle_mm_fault+0x28/0x2c)

[ 5507.342193] [<c0092a38>] (handle_mm_fault+0x0/0x2c) from [<c0014140>] (do_page_fault+0x14c/0x294)

[ 5507.342211] [<c0013ff4>] (do_page_fault+0x0/0x294) from [<c0008428>] (do_PrefetchAbort+0x3c/0xa0)

[ 5507.342232] [<c00083ec>] (do_PrefetchAbort+0x0/0xa0) from [<c0012678>] (ret_from_exception+0x0/0x10)

[ 5507.342241] Exception stack(0xcf839fb0 to 0xcf839ff8)

[ 5507.342253] 9fa0:                                     bec5ed44 00000000 00000001 00000000

[ 5507.342268] 9fc0: 0078ad58 000a2e42 00000007 00000001 000d2980 bec5ed3c ffffefc0 bec5f56c

[ 5507.342281] 9fe0: fbad2489 bec5ecf8 b6e4a218 000606e0 60000010 ffffffff

[ 5507.342289]  r8:000d2980 r7:00000001 r6:ffffffff r5:60000010 r4:000606e0

[ 5507.342305] Mem-info:

[ 5507.342312] Normal per-cpu:

[ 5507.342321] CPU    0: hi:   90, btch:  15 usd:  46

[ 5507.342340] active_anon:26812 inactive_anon:26852 isolated_anon:32

[ 5507.342340]  active_file:335 inactive_file:806 isolated_file:0

[ 5507.342340]  unevictable:0 dirty:0 writeback:1 unstable:0

[ 5507.342340]  free:4087 slab_reclaimable:566 slab_unreclaimable:964

[ 5507.342340]  mapped:62 shmem:109 pagetables:352 bounce:0

[ 5507.342340]  free_cma:0

[ 5507.342385] Normal free:16348kB min:16384kB low:20480kB high:24576kB active_anon:107248kB inactive_anon:107408kB active_file:1340kB inactive_file:3224kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:262144kB managed:251492kB mlocked:0kB dirty:0kB writeback:4kB mapped:248kB shmem:436kB slab_reclaimable:2264kB slab_unreclaimable:3856kB kernel_stack:840kB pagetables:1408kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:14118 all_unreclaimable? yes

[ 5507.342395] lowmem_reserve[]: 0 0

[ 5507.342404] Normal: 177*4kB (UM) 163*8kB (UEM) 106*16kB (UEM) 37*32kB (UE) 15*64kB (UE) 10*128kB (UEM) 4*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 2*4096kB (R) = 16348kB

[ 5507.342449] 1372 total pagecache pages

[ 5507.342459] 122 pages in swap cache

[ 5507.342467] Swap cache stats: add 59327, delete 59205, find 31106/31651

[ 5507.342473] Free swap  = 0kB

[ 5507.342479] Total swap = 131068kB

[ 5508.239017] 65536 pages of RAM

[ 5508.239040] 4310 free pages

[ 5508.239048] 2663 reserved pages

[ 5508.239053] 1164 slab pages

[ 5508.239059] 263096 pages shared

[ 5508.239065] 122 pages swap cached

[ 5508.239073] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name

[ 5508.239106] [  186]     0   186      568       49       5       15             0 systemd-journal

[ 5508.239120] [  190]     0   190      723        0       6       76         -1000 systemd-udevd

[ 5508.239133] [  194]   495   194      664       39       5        9             0 avahi-daemon

[ 5508.239146] [  199]     0   199      527       16       5       21             0 eventlircd

[ 5508.239159] [  206]     0   206      575       25       5       15             0 systemd-logind

[ 5508.239175] [  228]    81   228      694       69       5       14          -900 dbus-daemon

[ 5508.239188] [  263]     0   263      643       12       5        8             0 crond

[ 5508.239201] [  279]     0   279     1013       45       5       36             0 connmand

[ 5508.239214] [  320]     0   320     1176        0       5       60             0 wpa_supplicant

[ 5508.239227] [  321]     0   321      902       27       6       45             0 connman-vpnd

[ 5508.239241] [  330]     0   330      522       12       4       27             0 rpcbind

[ 5508.239253] [  336]     0   336     1065        0       5       67         -1000 sshd

[ 5508.239266] [  350]     0   350     3296       31       9       53             0 nmbd

[ 5508.239278] [  362]     0   362     3294       67       9       45             0 smbd

[ 5508.239293] [  410]     0   410     6770      361      13     2714             0 python

[ 5508.239307] [  441]     0   441     1769       44       6       53             0 sshd

[ 5508.239320] [  443]     0   443      644        0       5       28             0 sh

[ 5508.239332] [  457]     0   457     2276      285       7      623             0 python

[ 5508.239346] [  465]     0   465     1769       46       6       52             0 sshd

[ 5508.239359] [  470]     0   470      644       19       5       10             0 sh

[ 5508.239372] [ 3678]     0  3678   157710    52209     222    28131             0 xbmc.bin

[ 5508.239385] [ 3837]     0  3837      644       27       4        0             0 top

[ 5508.239394] Out of memory: Kill process 3678 (xbmc.bin) score 817 or sacrifice child

[ 5508.239412] Killed process 3678 (xbmc.bin) total-vm:630840kB, anon-rss:208800kB, file-rss:36kB

Milhouse · 2014-03-01, 22:30

I can even reproduce this "Movie library view" OOM behaviour by applying updates that don't result in any SQL updates. Here's a debug log.

My guess is that there's a cascade of Window/Dialog/GetDirectory calls, triggered in response to the JSON updates, even though there are no actual SQL updates applied.

cg110 · 2014-03-01, 23:13

Hmm, is that a bug somewhere in what I've done? Or that the slow code is unintentionally throttling the update events?

320 movies in 25s is 13 updates a second.
320 movies in 36m is 1 update every 9.4 seconds

Hmm, looks like it impacts windows as well. The cause is the JSON layer fires an update event for any item updates, and in doing so it's hit the main message loop with 300+ update events, which trigger major refreshes....

I think I've a fix for it, however, it's not something the code introduced, it was already there lurking in the shadows...

cg110 · 2014-03-01, 23:30

If you grab the head, specifically this commit:
https://github.com/cg110/xbmc/commit/514...a02d365104

I've added an api helper to push out an update of the fileitem, and so limit the scope of any updates to process. I'm hoping that reduces what gets updated to be targeted. Note that the message queue still gets hit with 300+ events though Sad

I've also added something to not send an update if nothing has changed...

Milhouse · 2014-03-01, 23:35

Yes, I'm not suggesting your code has a bug. Your code is now revealing this rather nasty problem as a result of being so much more efficient and generating so many more GUI updates than were previously possible!

cg110 · (This post was last modified: 2014-03-01, 23:42 by cg110.)

I should say I'd appreciate anyone that knows about the messaging api looking at the commit.
https://github.com/cg110/xbmc/commit/514...a02d365104

I couldn't find anything that said if param 2 should be a 1 or a 0. I did wonder if 1 is meant to indicate that the stream details have changed in some way as 1 appears to suggest it updates playlists, but it wasn't clear.

0 seemed a safe bet, and appears to work, IE the movie info display updates with updated values when the api changes them.

Or I wonder if:
CVideoDatabase::AnnounceUpdate

is a better option. Oh well off to watch tv and think about it...

Milhouse · 2014-03-02, 00:00

Just to let you know I applied the updated PR and the 320 updates now apply successfully in 33 seconds. However, as you predicted in the commit description, the GUI remains completely unresponsive for another 30 seconds after the last update has been processed.

Thanks, and enjoy your TV. Smile

**Montellese** · 2014-03-02, 09:31

The slowing down of updating the current view when lots of items are updated in a short time interval is the reason why we don't do this during a library scan. But there we actually know that we are probably going to get a lot of updates. With JSON-RPC we have no clue if there will be 1 or 10 or 100 or 1000 updates because every request/action is independent of the other.