SSH alternatives for mobile, low-latency or unreliable connections
console.dev
console.dev
Keithw (who's now a prof at Stanford) wrote mosh for that use case (high loss high latency), if memory serves correctly. (which it might not! being as apparently an entire decade has passed since "what feels like last month")
Also, there's some terminal cleverness, which I admit, to great embarrassment, I still haven't bothered to deeply understand (my "sensai", who offered to help me grok it if I ever got stuck, left this world too soon).
Mosh's ability to preserve interactive responsiveness, even through the most high latency, low bandwidth, high packet loss environments is unparalleled! You know, for all those times when you're on a cross country flight, your only network access is a IP-over-DNS tunnel, and you simply MUST ~zephyr~ chat with your friends. Uh, I mean, do mission critical work... ;)
I'm not sure what this author is talking about with tmux + mosh and adding the latency back in. Maybe because I run the tmux on the remote server and I mosh in. I don't miss native windowing or mouse controls at all, and still have scrollback. I modified the keybindings so it is easier for me (as a vim user) to remember -- including window splits, layout changes, and adjusting pane sizes.
I don't just use mosh in high latency, low bandwidth. The fact that mosh will reconnect even after I close the laptop (and put it in suspension), move from wifi network to cell and back, it's all fairly seamless. That makes it practical for my primary dev and ops environment on a remote server. I have had this setup for about 5 years now and use it every day.
I used to use something like autossh, and it did not work as well.
Making it more amusing, I know I suck at spelling so I even googled "sensai" first, before posting, quickly saw that it didn't have the "did you mean...?" prompt, and assumed it was the right spelling. Oops. :)
Turns out that it has been a github issue since 7 March 2012 with no progress or assignees: https://github.com/mobile-shell/mosh/issues/41
Anyone have a robust way of forwarding X11 from a Docker container from a remote host via ssh, even when an IP address changes, or a connection drops?
(Right now I'm typing this in a Firefox container running on Docker on a minimal virtual machine, ssh'd from my workstation, so I can use any version of anything without having to `trash` my workstation.)
I've used it on and off over the years, mostly when having to work on a fast train and a flaky 4G connection.
It is easy to future-proof the crypto for anything other than a quantum computer:
Ciphers chacha20-poly1305@openssh.com
KexAlgorithms curve25519-sha256@libssh.org
That is also the fastest cipher for any hardware lacking AES acceleration, but clients configured this way will never talk to older servers (or vice versa).File transfer is a real problem for SSH. The scp command is being slowly excised because of its security problems.
https://lwn.net/Articles/835962/
The replacement sftp protocol simply does not perform well, and the performance problems are explicitly not a development priority.
https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers...
Redesigns are on the table:
https://www.psc.edu/hpn-ssh-home
At this point, if someone baked the chacha20 cipher into vsftpd (replacing TLS), I would take it for internal use.
Wireguard is also an option, allowing us to simply use telnet and cleartext ftp (promiscuous mode on localhost would be a potential path to abuse).
In any case, people where I work have standardized on scp and sftp for communication between many disparate architectures, and few realized the problems this would bring.
It truly is the best, no? My pipelines look a bit different than yours (they usually have `lzop`/`zstd`, `cd`, and occasionally `pv`), but tar truly is the best file copy utility.
I rarely use `cp -a` even when copying locally. `tar` is so much faster.
$ lftp sftp://hostname
lftp ~> cd somedir
lftp ~> mirror .
lftp ~> quitNot only it works much faster than scp or sftp, but it also does not lose any file metadata.
I have not verified if newer ssh versions have corrected this, but a few years ago scp and sftp failed to copy extended attributes and on some file systems they truncated timestamps.
The first is the license. Because rsync is under the GPL, it will never be bundled in the OpenSSH.com distribution. It is even missing from OpenBSD base (just confirmed, and loaded it with pkg_add - there are two versions, one with iconv).
So let's take Microsoft's implementation, which will forever lack rsync (as they haven't touched it in well over a year).
The file transfer agent needs to be under the same license.
When you run rsync over ssh, the local rsync starts the local ssh client, which connects to the ssh server, which runs the remote rsync, while the data is piped through the ssh connection.
rsync is available for most operating systems, but it is not usually installed by default. OpenBSD might be the only exception without a rsync package, but it has openrsync. I assume that it uses the same protocol so rsync to/from OpenBSD should work as well as to/from Linux, Windows, MacOS or any *BSD.
If you have the right to transfer files to a computer then normally it is very simple to install rsync, only for your own user if do not have admin rights.
In most cases, you can initially just copy the rsync executable file to the target computer using scp, and then do any other file transfers with rsync over ssh.
After I have abandoned both scp and sftp many years ago, I no longer had any problems with the performance or data integrity of file transfers over ssh.
Nor will OpenBSD.
The rationale for this decision is not technical, it is idealogical.
A technical solution to this problem is insufficient.
To be honest, I was unaware of this:
For maximum utility, it should be bundled with OpenSSH.
Because openrsync has a BSD license, it can be used on any other system that completely avoids GPL programs.
I use rsync+ssh a lot, and it doesn't exactly work perfectly on long-distance 10-100 gigabit networks: ssh has a buffering problem at high bandwidths which the maintainers won't accept patches to fix.
Using scp or sftp would limit the transfer speed to a value several times lower than rsync in any case.
On short distance 10 Gb/s Ethernet I did not see problems, but it does not surprise me that on long-distance links they appear.
In such cases, if full speed is desired, it is likely that ssh must be replaced with either IPsec or a tunnel over TLS.
I don't know anyone in the HPC world who uses "IPsec or a tunnel over TLS".
This is interesting, can this be the reason for why playing 720p video in Firefox run with ssh and X11 forwarding is fine bandwidth-wise (easily 1Gb/s) but it causes responsiveness to mouse/keyboard actions to go into abysmal 5-30s territory?
Any references to the ssh buffering problem out there?
Looking at my terminal history, scp for a single file, rsync (which you don't need to pass that --rsh flag to.. during the 21st century ;P) for anything that requires the ability to be interrupted and resume gracefully (which is most cases), and, in the rare case when I need speed above all, the "tar lzop ssh bash pipe" trick (I don't know if it has a canonical name) is correct.
Isn't this what we all do?!? If there's a better way, I don't see how, but would like to know.
In that rare situation, I typically use tab completion to find the remote file. However, I bet that would SUCK on a high latency or low bandwidth link, where sftp would be just fine.
Solid Explorer (Android app) connects via sftp to Linux machines just fine. While I use Termux and rsync to do backups and bulk transfers, Solid Explorer allows me to rapidly browse remote files.
But yeah Rsync is a good choice.
Back in 1999, in one of my first professional gigs, people were still using ftp for sensitive data for automated data transfers.
Rsync+ssh would be great, but enterprise IT is slow to change.
AWS has hosted sftp backed by S3, and it's going to stick around for a while.
We're moving customers from FTP to SFTP. Most recent one was a well-known Fortune 500 company, which uses it to exchange some ASCII flat files with our software.
Mostly it's for XML files or PDFs.
It's simple, it works and it's easy for support to poke and prod.
Come to think of it, I once even set it up at a job in 2009. We needed to digest huge amounts of flat files from clients with little tech savvy. I put FTP on a server, put that behind OpenVPN, then sent the customers a pre-configured OpenVPN client installer. Never broke.
Tangentially, all this mention of FTP being used by banks and financial institutions is inspiring some choice nmap scans to run... ;)
Um. TLS can do chacha20 just fine if that's what you want. In particular in TLS 1.3 you can insist on only doing TLS_CHACHA20_POLY1305_SHA256 if for some reason you really don't like AES or maybe block ciphers in general.
Yes, throwing up an stunnel in front of port 23 enables telnet-ssl, but this is a bit heavy for a small machine.
I actually have an emulated VAX running VMS 7.3, and the C compiler on this machine does not support 64-bit "long long" so AES-CTR is the best that it can do (and chacha20 won't compile here).
In this case, a version of chacha20 for K&R C would be very handy. As you can imagine, sftp for this machine has been a unique challenge.
It's the author's mistake, not whoever posted it here.
Which makes me question how much consideration should be given to one who doesn't know what latency is. :) Assuming the best, it's a typo, we're all human. On the other hand, it's been more than a month (article was updated 2021-09-22) and he still hasn't corrected it...
sshfs user@computer.company.com:/home/repo ~/repo
vi ~/repo/file.pyIt still feels like there's value in making this work simply outside of emacs so more programs can make use of the cached copies. Is there a more resilient alternative to `sshfs`?
One thing that I might miss with a local buffer is good syntax checking support (I use "flycheck" in emacs) -- it requires the server's software, and so composing local editing with server-based syntax checking sounds pretty hard to get working well.
This is indeed one of the biggest downsides to mosh, and is a very often requested feature. But I am pleased that this article framed the lack of scrollback as an explicit design decision, and didn't simply call it a "missing feature".
In my usage, and opinion of mosh, the remote terminal emulation is at least half the point. It lets you be functional with minuscule amounts of bandwidth, and makes your sessions more responsive when you have plenty of it. It's efficient (enlightened even...) to solve an engineering problem using the appropriate distribution of resources (bandwidth, cpu, memory).
A remote terminal that wasted resources sending things that would be invisible anyway strikes me as not just less useful, but really mediocre/lazy design.
(IMHO, of course!)
Having to use tmux isn't a hassle for me as I would be using it anyway, and it has the added benefit of being able to connect to my active sessions on another machine and pick up where I left off.
However, the big drawback of Termius is that in the free version it will terminate the connection after quite a short while as soon as you put the app in the background. They offer a subscription to have it not behave like this but I don't want to pay a subscription for that feature. (The subscription includes additional features as well, but I don't want any of those features.)
When I read about Blink shell now, I was kind of hoping that it might be a good replacement. I tried it out on my phone, but I am a bit bewildered about how to select text using Blink shell.
I am wondering if maybe Blink shell is mainly suitable for use on iPad, and not as great to use on iPhone. At least, my initial impression is that it is probably fine for use with a keyboard but that it is not made for touch interactions to the same degree that Termius is.
Edit: I figured out how to select text with Blink shell; double tap and drag [1]. Unfortunately it feels quite clumsy and awkward to me compared to Termius. Not just the motion itself, but also how the resulting selection is handled. I will keep trying to use Blink shell for a while though. After all, Blink shell is open source and so if I like it a lot I could maybe rewrite the touch interactions to be more in line with what I'd like them to be.
https://twitter.com/dewitt/status/1329104663975587840
I too remember the era of ssh connections feeling strangely robust. I don't know enough about networking to fully understand how this works though.
But if that's possible at a networking level and all we need is a stable IP, then could an appropriately configured VPN solve these issues?
I had hoped on draft-bider-ssh-quic [1], but unfortunately it was marked as "no longer active".
Not all software needs to continually change. There is the concept of software that does one thing, and does it well.
THAT BEING SAID... if your complaint is regarding the SSH agent forwarding, I might be convinced to maintain a fork with that, and only that problem being addressed. :) That one's personal to me, and I've been meaning to poke at it since... well before it was released to the outside MIT community.
Not to mention CGNAT doesn't have large enough memory to respect TCP keepalive, it will drop your connection in a heartbeat if you even look at the session funny. I only have mobile internet at home and it was a pain until I wrapped it up in wiregaurd permanently.
Using SSH is now bliss... TCP keepalive is respected, an idle prompt will actually stay alive for the full 2 hours, which is basically impossible on any consumer internet connection today. I can switch physical connections without worrying about it dropping too... even when packet loss is crazy high wiregaurd ensures SSH doesn't know enough to drop the connection. I no longer worry about losing connections enough to usually not bother with tmux until I actually need multiplexing.
My internet experience in general has improved ever since using wiregaurd.
What security would you recommend to run on an unreliable connection?
To be specific, I have IoT devices which connect to the cloud using MQTT-SN. We use MQTT-SN because the underlying connection is NB-IoT (required for high penetration into difficult to reach areas). The latency on NB-IoT is quite high, which can lead to problems with TCP timeouts - so we have been recommended by the network operator to use UDP not TCP, which in turn leads to MQTT-SN, as it supports QoS 0 (unacknowledged).
But we are also very aware of security issues - so how would you put something like TLS onto a UDP datastream?
I am very frequently logged into remote systems, running tmux, and multiple terminals. At the same time I have multiple window manager windows running multiple of these tmux sessions. Different types of window arrangement. Different types of clipboard. Different keybindings. Different support for focus follows mouse. It’s janky af.
This article alludes to an iterm2 specific solution to the problem. I would dearly love to figure out a general one for my window manager of choice.
Help.
It's about as transparent/native feeling as you can get, and it works really well.
I don’t follow. Is this user running mosh inside a tmux session rather than running tmux inside the remote shell?
I don’t see how tmux adds latency back in.
It's very much like eternal terminal, but requires no upfront setup apart from having screen installed on the server.
In practice, it means that every time my laptop goes to sleep, or loses wifi for an extended period of time, I need to do an interrupt sequence and re-start it. Not the biggest deal, but things would be so much better if there was a way to configure it to re-try every so often instead.
> mosh: Last contact 9:45:19 ago. [To quit: Ctrl-^ .]
I'm not sure what the use of this message is -- surely you would think after 9 hours, trying again would be worth it.
I do just use default settings (just always set up mosh server on my remote servers and forget about it), but is it possible that there's some timeout on the other end you could be experiencing? If for any reason you kill the session on the other side, then of course it doesn't work, but for me it's always magically connected again.
I've used Mosh for multiple years, always loved it, until recently when I started using tmux more. For some reason I had loads of problems with colours and the only solution I could find was to manually build it from source with some flags, or even potentially it was a fork or something like that. Just started SSH-ing again (usually have a stable and fast connection) and if I lose the connection, I'll just re-attach the tmux session anyway.
I actually use tmux in combo and don't see any color issues that I notice. I wonder if it's something about the colors in your tmux config?
I run tmux (locally, nesting another session on the Mac) but I don't think that would make a difference here.
I'm not aware of any special config, but I've been using this setup for several years with no major adjustments.
# tmux auto attach or create new session [ -z "$TMUX" ] && { tmux attach || exec tmux new-session && exit;}
those are, in my opinion, essential tools to know how to use.
What? Isn't guardian just an agent? You can use ssh forwarding without it.
Guardian does look nice but seems abandoned.