Option to have XBMC use more available memory resources [NOT XBOX Platform]

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
xexe Offline
Fan
Posts: 711
Joined: Sep 2008
Reputation: 1
Post: #1
This thread is for the discussion of a feature request before/if it is added to trac. This is relevant to the non XBOX versions.


When you install XBMC it runs like a normal program. In many cases that is fine but what if you have an HTPC dedicated to XBMC?

I run a fairly large video library (26MB on disc) and on win32 XBMC never uses more than 160MB of RAM. This machine is dedicated to XBMC and has 8GB of memory but currently there is no way to make XBMC break it confines and start hogging memory.

There are lots of areas that would benefit from further memory use but the two that jump out are caching of thumbnails (for slicker folder and library browsing) and caching of older video frames (for slicker RW and step back).

I cant comment this code wise but in theory it makes a lot of sense to me.

Comments/ideas?
find quote
Gamester17 Offline
Team-XBMC Forum Moderator
Posts: 10,523
Joined: Sep 2003
Reputation: 10
Location: Sweden
Post: #2
See feature request:
http://trac.xbmc.org/ticket/4640
and feature request:
http://trac.xbmc.org/ticket/4641

Wink

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
find quote
xexe Offline
Fan
Posts: 711
Joined: Sep 2008
Reputation: 1
Post: #3
Both of those tracs are very interesting indeed. However neither touch on non video caching and in fact both focus on ways to maximise memory usage efficiency specifically for video playback.

To augment these, some thought should be given to caching all elements in UserData and indeed other data such as pictures and music. My userdata for instance contains just over 20,000 tbn files and whilst no view within XBMC would ever load all these the natural process of using the library will load a sizeable portion of them. I propose a method whereby userdata that is loaded once resides in ram for the duration of the session or is reclaimed if actually required to due a memory shortage (this is the crux of my idea). Drilling down and then back up in library view could therefore be optimised and disk access and more importantly responsiveness to the user improved.

I also believe most of the discussion re video cache are concerned around pre-emptive caching of data required for future playback, be it a few frames or a few seconds of video that are about to be played. The video that has just been played is less valuable to be cached but still valuable enough to not be dropped unless it needs to be.
find quote
jmarshall Offline
Team-XBMC Developer
Posts: 26,221
Joined: Oct 2003
Reputation: 178
Post: #4
I can think of 2 easy gains:

1. Have a (long) timeout on texture unloading, rather than instant unloading.

2. Have a better thumbnail cache.

2 applies without needing much more memory - basically saves hitting the disk all the time to check for existence.

A patch would be most welcome. See guilib/TextureManager.cpp, and look for ReleaseTexture(). Essentially you want to have an additional variable to ensure it doesn't unload, then an additional function to unload any expired textures. See how the xbmc/GUILargeTextureManager.cpp does it for instance.

Cheers,
Jonathan

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


[Image: badge.gif]
find quote
xexe Offline
Fan
Posts: 711
Joined: Sep 2008
Reputation: 1
Post: #5
From what i understand of what you are saying this seems like an excellent idea. Should I trac it as a feature request for 9.1? There is absolutely no way I could create a patch myself.

fyi i am xe' on IRC and pretty much there all the time should any real time discussion be needed.
find quote
EvilDude Offline
Senior Member
Posts: 104
Joined: Feb 2006
Reputation: 0
Location: Sydney Australia
Post: #6
Why not create a ram drive and store your UserData folder in there? Or maybe even just the whole XBMC folder should be in a RAM drive. That way the app, any dlls it loads etc are all already in ram..
find quote
EMK0 Offline
Senior Member
Posts: 209
Joined: Oct 2008
Reputation: 0
Post: #7
would this improve the speeds at which the fanart loads up?
find quote
topfs2 Offline
Team-Kodi Developer
Posts: 4,093
Joined: Dec 2007
Reputation: 12
Post: #8
EvilDude Wrote:Why not create a ram drive and store your UserData folder in there? Or maybe even just the whole XBMC folder should be in a RAM drive. That way the app, any dlls it loads etc are all already in ram..

I tried the latter option on linux, move all XBMC to ram and the speedup isnt that great actually. Actually more of a waste of it, I thought it would do more but Smile

Anyway, the databases and covers and such (mostly everything in UserData) would probably gain alot from this but it would add on start and shutdown time, if not designed properly.

Another gain would be to pre-load images that might be needed, thinking mainly on Home of most skins, they have lots of big images. long release would work here aswell.

If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

[Image: badge.gif]

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
find quote
xexe Offline
Fan
Posts: 711
Joined: Sep 2008
Reputation: 1
Post: #9
I thought about the ramdrive myself but my userdata folder is 777MB so that would represent to much of a delay on boot up. Also even if i could make it work i then have to deal with cleanly copying the changed files back to disk on shutdown which would be a kludge and even if it did work would make XBMC super slow to shut. In general people can accept slower boot times but cannot accept slow shut times and tend to kill program either manually or an OS shutdown. This approach also doesn't scale well to extrmelly large collections.

I cannot see the advantage of pre-load other than tricking users into thinking some views load faster at the expense of slower booting. If this could be a background process it would likely work but it seems like the wrong approach to me.

I am not looking for faster initial load times of XBMC or the views but rather faster load times on subsequent visits to pages you have already loaded content for/been to.

This is why I am saying keep things cached rather than drop them where the possibility exist.
find quote
Cheekyboy Offline
Junior Member
Posts: 32
Joined: Jun 2007
Reputation: 0
Post: #10
xexe Wrote:I thought about the ramdrive myself but my userdata folder is 777MB so that would represent to much of a delay on boot up. Also even if i could make it work i then have to deal with cleanly copying the changed files back to disk on shutdown which would be a kludge and even if it did work would make XBMC super slow to shut. In general people can accept slower boot times but cannot accept slow shut times and tend to kill program either manually or an OS shutdown. This approach also doesn't scale well to extrmelly large collections.

I cannot see the advantage of pre-load other than tricking users into thinking some views load faster at the expense of slower booting. If this could be a background process it would likely work but it seems like the wrong approach to me.

I am not looking for faster initial load times of XBMC or the views but rather faster load times on subsequent visits to pages you have already loaded content for/been to.

This is why I am saying keep things cached rather than drop them where the possibility exist.


Are you sure you know where the delays you experience are? I have a large database for music and video and only really experience delays in the music library such as opening or returning to large library views which JMarshall has traced to in memory parsing/processing of the data for the selected view after its read from the database.
For video I dont really have any noticable problems but then the number of videos will always be far less than music tracks anyway
find quote
xexe Offline
Fan
Posts: 711
Joined: Sep 2008
Reputation: 1
Post: #11
Your right i am completely guessing that using more memory will improve GUI usability but its a fair assumption.

My system sits with well over 7GB of free memory at all times and since it is dedicated to XBMC and I am streaming HD files of over 10GB and I can hear the HDD churning when i page up and down through library I am pretty certain there are improvements to be made.

Or put another way if i have watched 2GB of a 4GB video file and XBMC is using 150MB of memory why did it drop the 2GB of data when i have >7GB of ram free.

Its all about XBMC using the assets available to it, nothing more.
find quote
althekiller Offline
Retired Team-Kodi Member
Posts: 5,002
Joined: May 2004
Reputation: 12
Post: #12
So if I've read this right the gist of the request is, "Please enact piss poor programming habits so that I can make use of this asininely huge amount of memory I bought to inflate my e-penis." I know the caching policy could be improved in some places, but keeping a ton of useless crap in memory "just in case" we need it later is plain dumb. I don't see a dedicated XBMC system even touching 2GB of RAM with the largest of media collections.
find quote
rwparris2 Offline
Team-XBMC Python Developer
Posts: 1,333
Joined: Jan 2008
Reputation: 2
Location: US
Post: #13
xexe Wrote:My system sits with well over 7GB of free memory at all times and since it is dedicated to XBMC and I am streaming HD files of over 10GB and I can hear the HDD churning when i page up and down through library I am pretty certain there are improvements to be made.

Or put another way if i have watched 2GB of a 4GB video file and XBMC is using 150MB of memory why did it drop the 2GB of data when i have >7GB of ram free.

I can understand keeping fanart thumbs & nfo in RAM if it's avaliable and if it produces noticeably quicker loading for large libraries, but why in the world would you want 2GB of a video you've already watched to remain cached in ram? Faster rewinding?

Always read the XBMC online-manual, FAQ and search and search the forum before posting.
For troubleshooting and bug reporting please read how to submit a proper bug report.

If you're interested in writing addons for xbmc, read docs and how-to for plugins and scripts ||| http://code.google.com/p/xbmc-addons/
(This post was last modified: 2008-10-24 21:42 by rwparris2.)
find quote
xexe Offline
Fan
Posts: 711
Joined: Sep 2008
Reputation: 1
Post: #14
althekiller Wrote:So if I've read this right the gist of the request is, "Please enact piss poor programming habits so that I can make use of this asininely huge amount of memory I bought to inflate my e-penis." I know the caching policy could be improved in some places, but keeping a ton of useless crap in memory "just in case" we need it later is plain dumb. I don't see a dedicated XBMC system even touching 2GB of RAM with the largest of media collections.

No need to be aggressive and inflammatory. If you have a point make it, don't rely on insults to try and make it for you. You could have just asked why 8GB. The box is recycled from another duty and I have no use for the RAM elsewhere.

Currently on my collection XBMC uses 150MB of ram. That is no where near even the 2GB mark you cite and for instance it reloads images images from disk every time you page down and up through the library. Reloading from disk something that was in ram a few minutes ago when there is copious amounts of ram free is inefficient.


rwparris2 Wrote:I can understand keeping fanart thumbs & nfo in RAM if it's avaliable and if it produces noticeably quicker loading for large libraries, but why in the world would you want 2GB of a video you've already watched to remain cached in ram? Faster rewinding?

Yes that would be one advantage. Again the crux is XBMC on a dedicated HTPC can use all the resources it has access to since nothing else except for the OS will be contending for it. SAMBA caches to ram just like this and i can stop a video and come back the next day and if SAMBA hasnt been used much i can play it again direct from RAM.
(This post was last modified: 2008-10-25 09:26 by xexe.)
find quote