WIP Faster SFTP with libssh2 (Bug #14038)
#1
Based on the bug 14038 I started looking into the problem and found a solution which involves changing the used library from libssh to libssh2. Code can be found on my github account with all the ugly intermediate commits: https://github.com/lorem-ipsum/xbmc
First part of the code is a restructuration which doesn't change anything on the old libssh code. Based is the transformation to libssh2.
So far the code is only tested on linux, would be great if some one with a corresponding system could check the code on Windows and OSX.

With that patch set I'm able to get approx 6MB/s over the internet connection, compared to 600KB/s with the previous libssh code

Why libssh2 instead of libssh?
SFTP has the problem of the chunk size being limited to 32KBytes. If only one package is requested at a time and the time to answer is high that limits the bandwith and makes streaming of HD videos impossible (independent of bandwidth). To encounter that problem it's necessary to send oportunistic multiple requests in parallel. This has already been done in libssh2 but not in libssh, thus the change.
libssh2 is BSD licensed and should run under Windows, OSX and Linux

Whats missing?
  • Agent support is not yet tested
  • Check under windows and OSX
Reply
#2
to add to the list - has to be checked on android and ios too and tools/depends system has to beadded for libssh2 for beeing able to build for those platforms.
AppleTV4/iPhone/iPod/iPad: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for Kodi: NFS (wiki)
HowTo configure avahi (zeroconf): Avahi_Zeroconf (wiki)
READ THE IOS FAQ!: iOS FAQ (wiki)
Reply
#3
I glanced over the commits and like what i see. I hope i can find some time to get libssh2 into the tools/depends buildsystem so we can compile test this for the other platforms.

reminder: osx/ios/atv2 needs xcode project adaptions, windows needs vs project adaptions and libssh2 packaging (and i bet a bit more i guess).
AppleTV4/iPhone/iPod/iPad: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for Kodi: NFS (wiki)
HowTo configure avahi (zeroconf): Avahi_Zeroconf (wiki)
READ THE IOS FAQ!: iOS FAQ (wiki)
Reply
#4
I have no clue about the build system so i haven't done anything there yet. Don't know about libssh2 on android and iOS.

As commented on github the current implementation doesn't work if caching with the current FileCache, so I need some work there.

Tested the code yesterday in real life environment (watching a full streamed movie) and found lags if datarate of the movie went up, so a caching mechanism will be required for streaming. Since the caching mechanism seems to be quite complex, are there any guidelines / documentation about the caching mechanism? Would it be possible to implement an own version of file cache just for ssh/sftp (wouldn't like to do that if not really necessary).
Reply
#5
Out of current principal - it has to work with the current cache. A special cache just for ssh/sftp will not be accepted. Why do you think we need something special here?

Beside that i know near to nothing about caching - not my area Wink
AppleTV4/iPhone/iPod/iPad: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for Kodi: NFS (wiki)
HowTo configure avahi (zeroconf): Avahi_Zeroconf (wiki)
READ THE IOS FAQ!: iOS FAQ (wiki)
Reply
#6
I haven't dug very deep into the code of the cache, but my assumption is that the cache requests data in small chunks, while the opportunistic loading requires continuous data transfers to work.
In a quick test the transfer rate droped to only a few kbytes per second with the cache making even SD movies unplayable.
Reply
#7
Have a look at http://wiki.xbmc.org/index.php?title=HOW..._the_cache - maybe this helps you a bit in finding the code parts involved here (by search the code for the xml tags for example).
AppleTV4/iPhone/iPod/iPad: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for Kodi: NFS (wiki)
HowTo configure avahi (zeroconf): Avahi_Zeroconf (wiki)
READ THE IOS FAQ!: iOS FAQ (wiki)
Reply
#8
if we switch libs, we also should use the opportunity to bring the autotools build system up to date and use package config to detect libssh2.
Reply
#9
+1 Smile
AppleTV4/iPhone/iPod/iPad: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for Kodi: NFS (wiki)
HowTo configure avahi (zeroconf): Avahi_Zeroconf (wiki)
READ THE IOS FAQ!: iOS FAQ (wiki)
Reply
#10
I tried to find a solution for the caching problem and ended up reimplementing parallel requests in libssh (1, not 2). Code can be found in my github repo in the parallel-libssh branch (https://github.com/lorem-ipsum/xbmc/tree...lel-libssh).
Problem is, that libssh2 doesn't return any handles and does the precaching automagically. libssh has a function which returns a request to a handle and another to read from that. Now there's a queue which currently has a fixed size of 20 elements.
The only way I got libssh not to cause race conditions while using that system was to use a session just for reading and another for all the other stuff. Maybe there's an easier way to do that. I added code to initialize libssh for parallel requests, which currently assumes usage of pthread (which is propably wrong on windows). As far as I understood the documentation that initialization would have been required previously, as well.

ToDos:
  • Fix queue size (either as config parameter, or better automatically if required)
  • Check under Windows, OSX, Android, iOS
  • Check if there's a way to run that code without using a seperat connection
  • (Optional) Build a proper implementation of CBufferQueue

Edit: Added note to repo/branch
Reply
#11
Since Gotham 13.0 is out now (thanks for that, great release) I would like to bring that patch in a state that it can be merged for XBMC 14.0.

There are some points I would require input from a experienced dev:
1. Is it ok I implemented an own buffer queue? I guess it could be used in several other places and I would finish it to make it a complete implementation, not only the functions I use currently.
2. SFTP (or more correctly SSH in the background) uses host keys to identify hosts. The old (and current) implementation asumes the key is correct if unknown, if one is known but doesn't match it silently fails (just a message in the logs). It would be great to have a GUI integration which a) asks a user if the key is unknown, b) prints a correct error message if the key doesn't match. Where do I have to look for something like that?
3. Is the general code quality ok or are there some evil/ugly/unwanted/whatever things?
4. (Optional/Later) For a proper implementation of SFTP it would be great to support plain key files without the support of the keyring and support for Windows and Putty (I don't know if libssh supports that in some way, have to check that).
Reply
#12
Just like to say that improvements to SFTP would be greatly appreciated Smile

And +1 to points 2 and 4. Better key handling and feedback.
Image [RELEASE] Metroid
Image [RELEASE] IrcChat
Reply
#13
Very nice Job lorem_ipsum. Would be great if your solution will be integrated in the next Windows beta version of XBMC or Kodi

I have tried using sftp (would be ideal for streaming) and came to the conclusion that all 1080 content isn't watchable (stutters). So I switched to a setup with a VPN connection but a working SFTP implementation would be awesome! Would love to see this 'patch' work under the Windows version.
Reply
#14
I recently found a problem with sftp caching which might be related to your problem with buffering. The issue can be found at http://trac.xbmc.org/ticket/15391 and a fix has already been merged into master. This means that your SFTP stream might not have been buffered due to this bug (unless you had set the buffermode to 1 in your advancedsettings.xml).

I also noticed that both libssh and libssh2 allow asynchronous reads, so it seems to me like there should be no reason to use libssh2 instead of libssh. Especially if libssh2 needs to be checked on all platforms before it can be used at all. You wrote that you reimplemented the parallel requests in libssh (not libssh2), but it wasn't entirely clear to me if you're now using libssh, libssh2 or if you're trying out both (in separate branches). If you're trying out both, i'm curious as to which one you think would be best.

The most common way to improve SFTP performance seems to be to implement something similar to the sliding window protocol. With correct window sizes, it should allow you to achieve the highest possible throughput for your connection. Essentially, the window should be large enough to fit the amount of data tha can be received in the time it takes for the first chunk to be read and output to the appropriate location (e.g. xbmc's own buffer) thus allowing the window to slide forward by one chunk.

There seems to be an an additional complication for using SFTP with this sliding window protocol. Since TCP already implements the sliding window protocol, you have 2 sliding windows (from SFTP and TCP) running on top of each other. It seems like this causes the lesser of the two window sizes to effectively be the window size for the entire transmission. This is explained in the following post by one of the authors of HPN-SSH:
http://stackoverflow.com/questions/88492...r-than-ftp
I'm not sure if it is relevant or even useable in this case, but it might be interesting to read nonetheless.

All in all, I'm looking forward to seeing your performance improvements merged into xbmc and I'm happy to help if you need it.
Reply
#15
Thanks for the hint with the streaming bug. I assume it helps, but cannot fix the original problem, because the serialized requests reduce the bandwidth, which cannot be solved by buffering. The current implementation uses synchronous read, which is leads to a bandwith limitation due to the roundtrip times of the single requests. The stackoverflow post is interesting and might be an additional improvement, but for the beginning a proper asynchronous implementation shold help a lot.

My current implementation uses a queue of parallel requests with libssh which results in something like the sliding window. If you're interested youre welcome to look into my code at github and test, comment (and improve) it. I would love to bring it into the next xbmc/kodi release
Reply

Logout Mark Read Team Forum Stats Members Help
Faster SFTP with libssh2 (Bug #14038)1