DEV Community

Reynold Chery
Reynold Chery

Posted on

Replacing OpenSSH/SFTP with SFTPGo

#go #openssh #systemd #linux

I run a pair of GNU/Linux VPS's, one running Plex as a client and the other as a storage server for media. They both run on "the cloud" hosted by different providers. I've been noticing lately that transfer rates between them over SSHFS have been lagging, causing buffering and drop issues for some of my media streams. I tested directly between the hosts using Iperf3 and while not spectacular between two large hosting providers, it's fine for streaming SDTV content.

me@plex:~$ iperf3 -c 10.10.10.100 -p 5201 --bytes 1000000000 Connecting to host 10.10.10.100 , port 5201 [ 4] local 10.10.10.99 port 56634 connected to 10.10.10.100 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 57.0 MBytes 478 Mbits/sec 165 3.94 MBytes [ 4] 1.00-2.00 sec 45.0 MBytes 378 Mbits/sec 17 1.99 MBytes [ 4] 2.00-3.00 sec 43.8 MBytes 367 Mbits/sec 0 2.02 MBytes [ 4] 3.00-4.00 sec 42.5 MBytes 356 Mbits/sec 0 2.15 MBytes [ 4] 4.00-5.00 sec 48.8 MBytes 409 Mbits/sec 0 2.41 MBytes [ 4] 5.00-6.00 sec 52.5 MBytes 440 Mbits/sec 0 2.76 MBytes [ 4] 6.00-7.00 sec 60.0 MBytes 503 Mbits/sec 0 3.02 MByte
Enter fullscreen mode Exit fullscreen mode

I then moved on and tested the mount point. I mount the media storage filesystem from plex client to storage server using SSHFS. So I performed a crude check on the SFTP mount point using cat and a standard utility called pipe viewer (pv).

me@plex:/mnt/media/tv/show/S01$ cat show.mkv | pv -rb --progress >/dev/null 4.88MiB [ 300KiB/s]
Enter fullscreen mode Exit fullscreen mode

indeed, file read hovers around ~300KB/s (with multiple stalls) were just bad. Now there are a slew of tweaks that can be done, via the ssh protocol lighter MACs, Ciphers, via SSHFS buffer cache, VFS tweaks etc.. and the network, buffer, window sizes etc.. I decided to sidestep all the tweaks and try a different SFTP implementation entirely. I remembered a hacker news post not too long ago and decided download a binary release of SFTPGo.

SFTPGo is a standalone SFTP server written in Go, that does not rely on the system authentication mechanisms and can utilize multiple SQL authentication back-ends such as SQLite, Mysql etc.. but I wanted a quick way to use public/private key authentication and good thing it supports this via its "portable" option.

First we'll create an ssh public/private key pair on the Plex client VPS

me@plex:~ ssh-keygen -t ecdsa -f ~/.ssh/id_ecdsa_sftpgo
Enter fullscreen mode Exit fullscreen mode

We then add a section in ~/.ssh/config on the client that describes the storage server

Host storage_server Hostname 10.10.10.100 Port 4444 User sftpdude IdentityFile ~/.ssh/sftpgo_ecdsa Ciphers aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com MACs umac-128-etm@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1,hmac-sha2-512-etm@openssh.com,hmac-sha2-512
Enter fullscreen mode Exit fullscreen mode

Now we'll move to the storage server and create a dedicated directory and download the latest release of SFTPGo and install it manually

me@storage:/ sudo mkdir /opt/sftpgo me@storage:/ sudo chown me:me !$ me@storage:/ cd !$ me@storage:/opt/sftpgo wget -O - https://github.com/drakkan/sftpgo/releases/download/0.9.6/sftpgo_0.9.6_linux_x86_64.tar.xz | tar Jxf -
Enter fullscreen mode Exit fullscreen mode

Next create a wrapper script on the storage server and copy in the client's public key that was created earlier in ~/.ssh/id_ecdsa_sftpgo.pub

me@storage:/opt/sftpgo/ cat sftpgo_start.sh #!/bin/sh sftpgo=/opt/sftpgo/sftpgo port=4444 pubkey='ecdsa-sha2-nistp256 AAAAE2V....3YPCHWceWD2QcQFG=' dir=/mnt/media $sftpgo portable --username sftpdude --public-key "$pubkey" --sftpd-port $port --directory $dir --permissions '*'
Enter fullscreen mode Exit fullscreen mode

We run the newly created script on the storage server

me@storage:/opt/sftpgo/ /opt/sftpgo/ssftpgo_start.sh
Enter fullscreen mode Exit fullscreen mode

We connect to the storage_server via sshfs

me@plex:~ /usr/bin/sshfs -f storage_server:/tv /mnt/media/tv2 -o ServerAliveInterval=15 -o rw,reconnect,idmap=user
Enter fullscreen mode Exit fullscreen mode

and retry via the crude cat | pv

me@plex:/mnt/media/tv2/show/S01$ cat show.mkv | pv -rb --progress >/dev/null 23.8MiB [2.84MiB/s]
Enter fullscreen mode Exit fullscreen mode

and lo and behold .. much faster transfer rates without tweaking and just switching implementations. Of course it's far slower than a raw byte transfers (iperf3) but we are utilizing FUSE which in itself is a bottleneck traversing user-space/kernel boundaries, but it's good enough for this use case.

Securing things

OpenSSH has decades of security under its belt and comes from the iron clad OpenBSD folks and we want to protect the system as much as possible from any weird side effects, bugs, and/or exploits typical of long running programs. While it is written in a garbage collected language (Go) which eliminates many classes of memory induced bugs, it is better to be safe than sorry. So if you're running a popular Linux distro today you most likely have systemd installed by default. Systemd is an initd/framework for starting/managing the system and it's services. It is what we'll use to to monitor, manage and secure the SFTPGo service. First thing we do is think about that what the service needs to do to perform its job; it starts up, creates a pub/priv key file if /opt/sftpgo/id_ecdsa doesn't exist, listens on a specified port, and reads/writes to /mnt/media/. So let's create a systemd service file that incorporates this behavior.

So here's the systemd service file

me@storage:/etc/systemd/system cat sftpgo@me.service [Unit] Description=SFTP Go AssertDirectoryNotEmpty=/mnt/media/ [Service] User=%i ExecStart=/opt/sftpgo/sftpgo_start.sh WorkingDirectory=/opt/sftpgo ExecReload=/bin/kill -s HUP $MAINPID KillMode=mixed Type=simple Restart=on-failure RestartSec=10 StartLimitInterval=20s StartLimitBurst=3 NoNewPrivileges=yes PrivateTmp=yes PrivateDevices=yes DevicePolicy=closed ProtectSystem=strict ProtectHome=yes ReadWritePaths=/opt/sftpgo /mnt/media/tv  RestrictAddressFamilies=AF_INET #requires systemd 235+ and kernel 4.11+ IPAccounting=yes IPAddressDeny=any IPAddressAllow=10.10.10.0/24 [Install] WantedBy=multi-user.target Alias=sg.service
Enter fullscreen mode Exit fullscreen mode

Basically we prevented read/write access anywhere except /opt/sftpgo and /mnt/media/tv. We restricted access to /home or any other filesystems via the ProtectHome, ProtectSystem stanzas. We also restricted socket creation to the AF_INET protocols while only accepting connections from Plex client ip address range. Systemd provides a host of facilities one can use to secure their services without that much effort in tandem with the kernel 4.x+ namespacing, eBPF, cgroup controls. Most of systemd's lock down parameters are explained here, there's even more we can do such as system call, namespace filtering, but we'll need to profile the binary some more before implementing that.

Now we enable and start our new service

me@storage:~ sudo systemctl enable sftpgo@me me@storage:~ sudo systemctl start sftpgo@me me@storage:~ sudo systemctl status sftpgo@me
Enter fullscreen mode Exit fullscreen mode

And check some transfer stats after transferring some files (since we enabled IPAccounting=yes)

me@storage:~ sudo systemctl show sftpgo@me -p IPIngressBytes -p IPEgressBytes IPIngressBytes=20382896 IPEgressBytes=677305852
Enter fullscreen mode Exit fullscreen mode

99% of the time I have no need to replace the standard SFTP server implementation on my Linux/*BSD hosts, but it was worth it this time for some flexibility and the bump in speed. Though the OpenSSH SFTP implementation is actually faster than SFTPGo, so clearly something is wrong between these hosts that I need to troubleshoot down the road. But, just the fact that I can decouple the standard GNU/Linux user authentication mechanism from SFTP and name the user whatever I want as long as the script has the correct read/write permissions is a plus for me.

Top comments (0)