Linux - Buffering on SSHFS mount

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #1
Hello,

I'm using XBMCbuntu 11.0 (which is running XBMC 11.0 Git:Unknown - Compiled Mar 24 2012) on a Linux 3.0.0-23-i686

My videos are stored on a remote filesystem mounted at startupt (in /etc/network/if-up.d/) using SSHFS.

As my internet connection is not as fast as I would hope, I tried to increase the buffer to 64MB in the ~/.xbmc/userdata/advancedsettings.xml file (I also set loglevel to "debug") :
Code:
<advancedsettings>
  <loglevel>1</loglevel>
  <network>
    <cachemembuffersize>67108864</cachemembuffersize>
  </network>
</advancedsettings>

The log file indicates that it is successfully loaded :
Code:
15:16:30 T:6633216  NOTICE: Loaded settings file from special://profile/advancedsettings.xml
15:16:30 T:6633216  NOTICE: Contents of special://profile/advancedsettings.xml are...
                                            <advancedsettings>
                                              <loglevel>1</loglevel>
                                              <network>
                                                <cachemembuffersize>67108864</cachemembuffersize>
                                              </network>
                                            </advancedsettings>

My problem is that the configuration does not seem to be applied : xbmc's memory usage is not bigger when using this conf (so I'm supposing nothing is cached further than the default 5MB).
To verify that assumption, when watching a video (720p scene rip) and pressing the 'o' key to display details, the end of the 3rd line says :
Code:
cache: 0 B 100%
The percentage value is of course moving depending on the cache usage, but it's pretty clear that when the video is freezing, the percentage value goes up to 100% within less than 10 seconds, which is absolutely not enough to fill up 64MB of cache.
Also, when I pause the video, the cache is not filling up at all (the percentage value is not moving and nload indicates that there's absolutely no network activity).

So basically, I can't really watch a video because it's freezing every 2 minutes to fill up the seems-to-be-very-small cache. So here are my questions :
1. am I doing something wrong when increasing the buffer size ?
2. is extra buffer size actually used when working on a SSHFS mount (as it's seen by XBMC as a local filesystem) ?
3. why do I see "LogLevel changed to 1" in the log, but not "cachemembuffersize changed to 67108864" ? (see full log attached)
4. I've seen the new latency/delay option (http://wiki.xbmc.org/index.php?title=Use...3Cvideo.3E) but I'm not sure what it does and/or if it can help me. Any hint ?
5. any idea what else I could try ?

Thanks in advance for any help, I've spent almost the whole night trying to figure out how to do it.

PS : XBMC log can be found here : http://pastebin.com/i2JutQVC (i've just removed all the VideoInfoScanner and GetImageHash lines with all the filenames and stuff, I just left the one I used for the test : Breaking Bad S05E01)
PS2 : if you need any more information, please let me know
find quote
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #2
Ho, by the way, I forgot to mention that the problem is exactly the same whether I use wired or wireless network connection.
find quote
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #3
Or maybe there's an add-on somewhere out there to "preload" movies ?
find quote
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #4
Bumping, just in case...
Am I the only one to use SSHFS ?
Is there a different buffering system when using NFS, for example ?
find quote
.:B:. Offline
Fan
Posts: 424
Joined: Mar 2010
Reputation: 0
Post: #5
I believe that cache setting is some legacy stuff from the Xbox days, apparently there's no way to set the buffer with Xbmc for networked file systems.

If you are saying 'remote' and 'internet connection', do you mean your share is really somewhere online, not on your LAN? You need quite a bit of bandwidth to stream stuff... Especially if the video has not been optimised for streaming.

* Netgear WNDR3700, OpenWrt Barrier Breaker 14.07+ git :: HP Procurve V1410-8G switch
* OpenELEC 4: HTPC Zotac Nano AD10 with Pulse-Eight CEC adapter, Sony KDL52-NX800 :: Desktop Phenom II X4 980 BE / AMD HD6950 / 23" Dell Full HD
* Debian Jessie x86_64: Celeron G1610, HTTP/NFS/MySQL/ZFS server
* Blog
find quote
wsnipex Offline
Team-Kodi Member
Posts: 3,836
Joined: Jun 2011
Reputation: 94
Post: #6
the buffer will only be used, if xbmc regards your source as "remote", which is not the case with a FS mounted in the OS.
afaik it only works with stream URLs. There is a PR that enables buffering also for local mounts, but its not in mainline yet and definately not in Eden.

see here: https://github.com/xbmc/xbmc/pull/831
find quote
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #7
(2012-08-10 01:14).:B:. Wrote:  If you are saying 'remote' and 'internet connection', do you mean your share is really somewhere online, not on your LAN? You need quite a bit of bandwidth to stream stuff... Especially if the video has not been optimised for streaming.

Yes, my share is "over-the-internet", as I have much more (10x) disk space on my server than I have at home... and my internet connection is quite shitty (4Mbps down) but it's usually enough to downlaoad a small-sized (DVDRIP and well-compressed 720p) movie/episode within the duration time of the said movie/episode.
But it's blocking when watching not-so-compressed 720p or even 1080p videos (or when the video has high-bitrate scenes).

So I'd love a buffer to download (= read through SSH) maybe 5 minutes of video prior to reading it, even if I must wait 5 minutes at the beginning of the video for the buffer to fill up.
(2012-08-10 09:02)wsnipex Wrote:  the buffer will only be used, if xbmc regards your source as "remote", which is not the case with a FS mounted in the OS.
afaik it only works with stream URLs. There is a PR that enables buffering also for local mounts, but its not in mainline yet and definately not in Eden.

see here: https://github.com/xbmc/xbmc/pull/831

Thanks for answering my question (about whether the buffer is used with SSHFS mount or not) and for pointing me to this PR, it seems interesting !
So if buffer is use only with stream URL, does that mean you can't read a whole directory tree, but only one file at a time ?
Or is the buffer used when declaring a SMB or NFS source from within XMBC (not mounting it using /etc/fstab or regular `mount` command)
(This post was last modified: 2012-08-10 10:17 by gornack.)
find quote
wsnipex Offline
Team-Kodi Member
Posts: 3,836
Joined: Jun 2011
Reputation: 94
Post: #8
imho its not used at all without this PR unless you use http, rtp, udp etc streaming.
You could try to stream using one of those from your server instead of ssh.

Or compile from source with the PR applied and retest.

edit: what do you mean by "read a whole directory tree"? xbmcs movie player(dvdplayer) only reads the file it is currently playing.
(This post was last modified: 2012-08-10 11:15 by wsnipex.)
find quote
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #9
(2012-08-10 11:13)wsnipex Wrote:  imho its not used at all without this PR unless you use http, rtp, udp etc streaming.
You could try to stream using one of those from your server instead of ssh.

Or compile from source with the PR applied and retest.

edit: what do you mean by "read a whole directory tree"? xbmcs movie player(dvdplayer) only reads the file it is currently playing.

Ho sorry it was not clear. By "read a whole directory tree" I meant "be able to browse my files". I don't see how http or rtp could let me browse my files (they are protocols used to remotely read 1 file).
find quote
Stilren Offline
Junior Member
Posts: 1
Joined: Oct 2012
Reputation: 0
Post: #10
Hey,

This answer might be a tad late, though in case you still haven't solved this issue:

I had the same problem with a lot of different compilations out there, however I knew for fact that this was not due to my internet connection speed.
I'm guessing that it should be fine with your connection as well.

To make a long boring story short, it works for me with the Openelec Release: 1.0.2 (1.0 Final). Any other XBMC would buffer insanely often on 1080, though most of them played 720 series etc. perfectly fine.

Edit: And they would only buffer over SSHFS

Regards,
S
(This post was last modified: 2012-10-05 15:36 by Stilren.)
find quote
gornack Offline
Junior Member
Posts: 7
Joined: Aug 2012
Reputation: 0
Post: #11
Hi Stilren,

Thanks for your answer.
I actually found a pretty neat solution in the pull request : https://github.com/xbmc/xbmc/pull/831
find quote
Johan84 Offline
Junior Member
Posts: 5
Joined: Mar 2009
Reputation: 0
Post: #12
(2012-08-10 09:02)wsnipex Wrote:  the buffer will only be used, if xbmc regards your source as "remote", which is not the case with a FS mounted in the OS.
afaik it only works with stream URLs. There is a PR that enables buffering also for local mounts, but its not in mainline yet and definately not in Eden.

see here: https://github.com/xbmc/xbmc/pull/831

Hmm, I have been struggeling with the buffering/advancedsettings during at least the last year.
The common thing is everywhere I use XBMC, I use VPN+SMB/NFS.

So you are telling me, that since I use VPN, and mount SMB, my connections are treated like LAN, and I won't get any caching done?

That seems right, because no matter what I set the buffersize to, the videos start playing instantly and I don't see network usage that should look like buffering (e.g, on a network capable of 1 megabyte/sec, I expect to see 50 megs of traffic the first 50 seconds in order to fill a 50 megabyte configured buffer).

So... how to I solve my vpn+smb buffer problem?
I want XBMC to:
1. Fill my 50-100 meg buffer asap, with a download of 1meg/sec, I expect constant traffic for 50-100 sec
2. I then want XBMC to constantly try to have this buffer filled

Problem is, I dont see that behaviour, its more like
* 0 kB/sec
* OMG OMG OMG out of buffer need to get some data, 1 meg/sec
* 0 kB/sec
* omg omg get data 1meg/sec

And so on... I'm no video/codec guy, but I have big problems understanding the logic in how data is fetched since I almost always get "buffering" problems when I'm at a remote location doing VPN. But there's no problem with the connction:
Code:
rsync 330 meg file, over the same VPN/smb
sent 334767911 bytes  received 42 bytes  990957.23 bytes/sec


I'm trying to play a 42 min 720p file, almost exactly 1 GB in size, I guess this would give me and average bitrate of something like:
1000 megs / 42 mins / 60 secs ~= 0,4 megs / sec.

So.. my connection has more than double the necessary speed, but still i get the damn BUFFERING.


Even with a REALLY simple "strategy" for "buffering" in XBMC it should work well:
1. Use ALL available network speed to start downloading the movie to e.g /tmp
2. Start playing the movie
3. With a 42 minute file, 1000 megs of size, 1 meg / sec download, the movie would be all downloaded in 1000/1 = 1000 secs / 60 secs = 16 mins... well before the movie has ended.

And this is actually what I'm doing nowdays...
I start copying the file from the remote VPN smb share, and start viewing it with VLC.
Since the copy moves at full speed at all times, I have no buffering issues and the copy is done after arount 16-20 mins, well "buffered" out before the movie has ended.

I hope my reasoning isn't too unclear.. Tongue
find quote
wsnipex Offline
Team-Kodi Member
Posts: 3,836
Joined: Jun 2011
Reputation: 94
Post: #13
I see 2 options for you atm:
1. compile yourself with the buffering patch applied
2. switch to a streaming protocol instead of smb.
find quote
Johan84 Offline
Junior Member
Posts: 5
Joined: Mar 2009
Reputation: 0
Post: #14
(2012-11-18 18:07)wsnipex Wrote:  I see 2 options for you atm:
1. compile yourself with the buffering patch applied
2. switch to a streaming protocol instead of smb.

I switched to webdav, and if I press 'o' I can see how it fills the buffer to 20 megs, and then does exactly what I am expecting, keeps trying to hold the buffer at 20 megs at all times Smile
So frustrating that I have been trying to "solve" this issue during the last year (!), and the "solution" was there all along, nfs/smb does not even buffer Tongue
Just hoping that the buffering patch will get pulled into the main build!

Thanks a lot for the help!
find quote