Mosh
jefftk.com
jefftk.com
When this is not so appealing to desktop users, it is a godsend feature for laptop/mobile users. Especially, if you are an iPad user you should try Blink shell app [1] with Mosh. This combination turns the iPad your favorite portable terminal machine.
I was already using tmux for session persistence, so the local echo and handling of congested networks were the two features that made me wish I'd switched to mosh earlier.
(I use Prompt 2 on my 12.9’’ iPad Pro which doesn’t support mosh. I only use mosh on computers with very high latency servers.)
I have to admit that terminal is inherently not suited for touchscreen. Here I am suggesting iPad Pro a laptop alternative for a dedicated terminal use, where you use a physical keyboard.
Btw, tmux scrollback sucks except when the terminal emulator's native scrollback is supported through control mode.
I almost switched to mosh in the past while searching for this kind of solution to short-living ssh connections.
But I've finally never made the jump, re-launching ssh/tmux being four keystrokes only (up+enter x2). Not justifying changing the whole paradigm. Plus you get the additional bonus with tmux to be able to recover the same remote terminal sessions when switching computers (leave stuff in place at work, go home, continue on the same stuff at home).
Edit: maybe it's the time for jumping ships!
ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 $HOST
Suppose I’m doing some remote debugging stuff on my way to work. I have one IP on my home Wi-Fi. As I walk to my bus stop, it switches to LTE. As my bus drives through the city, I periodically jump to different addresses. I grab a coffee on my walk to the office and end up on the coffee shop’s Wi-Fi with their address. I leave and hop back to LTE, until I get to work and connect to its Wi-Fi.
With SSH, every one of those changes means reconnecting. With Mosh, it means that everything pauses for an instant and then resumes as if nothing ever happened.
Advantages:
- you dont see any lag when you type, it is exactly as if you were on your local machine
- when network disconnect, the terminal freeze and when network is up again, the terminal becomes interactive again preserving everything in between
- connection is preserved as long as the terminal is open, so even after hibernation your session is preserved
Drawbacks:
- you can't scroll , but if you use tmux then it is possible . I've tried using Eternal terminal, as it allows to scroll but it doesn't perform well at all on laggy connections
- You can't detach and re attach mosh connections (same for ET)
- I'm not sure it is possible to preserve middle-click copy/paste (at least I've not managed to make it work)
- time to connect is a bit longer than normal ssh
The lag that is eliminated is the local echo of characters as you type, as mosh is sending those asynchronously in the background. If you have fancy, more interactive prompt/cmdline on the remote server then you'll still see the issue.
Ditto if you run tmux on the remote server. I really wanted to like/use mosh, but there's too many issues of reality that it can't gloss over.
If you're just maintaining basic SSH sessions into remote servers, then it's great. But if you have a nice, interactive remote development environment then you still get the same lag and latency issues.
The lag is something that I can deal with. For me, the killer feature is the automatic reconnects. I usually only use mosh when I’m traveling and at conferences. In that scenario, I’m either using my iPad (with keyboard) where the client might be killed anytime I switch apps or I’m using the conference WiFi, with is always spotty. In either case, using a mosh client makes me less likely to throw my laptop or iPad down in anger every 5 minutes.
I'd say this is a perfectly fine solution for a lot of cases. Not all, of course, like your iPad one sounds a great fit for mosh (and I'm gonna try it out!) But even a RaspberryPi can power a really powerful terminal.
You can use sshfs[0] to mount a remote drive over ssh. Obviously there will still be some lag - `ls` will require a round trip - but your local terminal will be much snappier.
How is mouse support in Blink? I've been searching online and can't find a good answer. I regularly use the mouse in Terminal.app to manipulate Tmux windows and move around in the Micro text editor. I am curious about Blink, but hesitant to drop $20 if it turns out it doesn't work the way I need it to for terminal usage.
But Blink is the app I use for SSH/mosh on the iPad too.
That basically describes TRAMP mode, right? It works well.
- No commits in github in over 9 months, PRs and issues are stacking up.
- I've had friends report clipboard broken in latest Mac OS X version.
Don't get too attached to mosh.
Why do you suffer to use a local server/internet for development? I've been using the "laptop is a dumb terminal" paradigm for many, many years and I've avoided a lot of headache that I've seen in others trying to get everything to work "locally".
Yes, this means I need to be online to work, but almost nobody works offline anymore - between googling issues, downloading/updating dependencies, and browsing while compiling, having an internet connection is assumed.
Remote development is tricky indeed. Many people run the IDE locally and use its remote development functionality. I run it remotely over VNC.
I've been wanting to try out remote developing for multiple years now, it would allow me to get an iPad with a perfect battery life or an Macbook Air that would tag along all day without charging. But because I cannot work with vim/emacs and my IDEs (IntelliJ) do not support anything like remote development I am stuck with a huge Macbook Pro including a battery draining IDE that cannot be ran remotely. VNC added too much lag for me to use all day.
I feel like this is the best of both worlds- but it does require some up front effort to build the pipeline
It didn’t even occur to me that this was a foreign concept to folks!
As someone who’s been fairly mobile/nomadic most of my career, I have always enjoyed the thin client approach.
I can get many more years of service from my client hardware when offloading the bulk of compute, storage, and stable bandwidth needs to beefy servers.
I’ve found that it’s just generally more economical and efficient this way.
Plus, the few times I’ve had laptops die, get stolen, damaged etc to have a much lower impact when they don’t have anything all that important on them.
Using VSCode in this way makes you much less dependent on good SSH. When using emacs in a terminal over SSH, your whole UI is being rendered over SSH and any latency or packet loss is going to affect UI feedback and rendering.
If you use VSCode Remote-SSH, you UI is rendered locally, so typing and editing work smoothly, and you don't really notice a bit of network latency on a request to "find in files" for instance.
Running your emacs locally and then using Tramp to edit remote files (as suggested elsewhere) gives you some of that benefit but is a much more limited development experience, and (in my experience) flakier.
Developing a python framework for running system tests on hardware. This can only be run on the machine that is in the lab connected to the hardware. For various reasons related to the network latency of being at home, the easiest way to iterate on that is to checkout and develop on the machine itself.
Developing on Linux, from home. My work laptop is a (small) Windows machine. Normally it doesn't get heavy use - some presentations and email reading when I travel. Now I'm WfH it's my primary machine. By SSHing to my Linux desktop at work (for terminal and for VSCode), I get the development experience of Linux on my Windows machine (there are some other benefits to do with that machine being on the right network and close to the stuff it needs to talk to a lot).
Developing FPGA code. This requires a hugely powerful machine to compile on if you want a reasonable compile speed. The Hardware group have a few specialised machines for this that are shared, so that we can have laptops and get the benefit (also means you can leave a synthesis going overnight/over the weekend easily).
(If you already know, well, I'm sure someone else just learned something today.)
The issue I ran into with editing code through tramp is the integration with make. I use `M-x compile` to compile from emacs, with a few modifications to find the appropriate makefile depending on which project I am working on. If I try to compile locally based on remote files, my compile times go through the roof. I suppose I could have opened an ssh session in which to run make, but it was easier to open emacs remotely.
The other advantage is that I could use tmux+emacs to have everything remote, and to connect to that same session from multiple computers. A typical day during an experiment might have me start in an office, later head to an experiment monitoring room, and later head down to the experiment vault to see if an issue is being caused by hardware. With remote editing, I could have the the entire session, including any files that I have already opened, no matter where I am physically located.
Docs are here: https://www.gnu.org/software/tramp/#index-compile
- If you're walking the filesystem, make sure you start from the "default-directory" variable.
- If you call other programs, make sure you use "process-file" (for synchronous processes) or "start-file-process" (for async). Using "call-process" or "start-process" unfortunately won't do the right thing, but the two functions I mentioned are mostly drop-in replacements.
Unfortunately, my current job doesn't use Linux as much, and so it has been too long since I've spent time improving my .emacs file, as it is only home projects at this point.
Also, security policies. But I like the first reason more.
Then again, my laptop won't run friggin mosh, because of security policies.
This can be all done in a VM, of course. I personally like the remote server, especially since they can be spun up so easily with cloud services now.
[1] - Though sometimes that's impossible. I.e. embedded.
ADD: I do switch frequently between my desktop and multiple notebooks. Working this way means I can have a beefy/noisy development server and a light & quiet notebook which just needs mosh.
Also, I have a mosh client on my smartphone; setting up the dev environment there would be a bear. Essentially the mainframe model ;)
I switched from Mosh to et across 12ish servers around 6 months ago and am way happier for it.
Previous conversation:
Thats the trade-off. The "local echo" feature is what I use mosh for as I deal with system 8 to 10 timezones away. Doing this breaks scrollback.
I guess when mosh was first made, this was a problem with ssh shell sessions over EDGE or 3G as well, although network latency is less of a concern these days on 4G LTE or 5G mobile.
mosh server -- screen -xRR -D
Here some medium article about this -
https://medium.com/@toja/tip-using-mosh-with-scrollback-257a...
On top of that, I use both tmux and Mosh at the same time. You don't need to use tmux exclusively.
I meant, you don't need to get scared. Both ET and Mosh built on top of SSH and there never be critical errors when dealing with their shell. Though both of them requires installations on the remote but they are a single zero-config packages: just install a package (not "a bunch of") and that's it. No config needed, even no background systemd services needed.
what difficulties are you facing there? i use tmux deeply nested without problems. one trick i use is to change the tmux command key to be something different on each nesting level.
This is exactly the difficulty I never want to see...
And I what I said that the problem was the moment I have to have different tmux config for every machine, just like she/he is doing.
And when you're setting up a server it's easy to turn off the tmux status bar.
the multiple toolbars don't bother me either. they can serve as a visual aid to recognize where i am. each terminal looks different. but in reality i don't pay much attention to them.
[1] - https://github.com/mobile-shell/mosh/issues/337
[2] - https://www.bountysource.com/issues/4471419-ssh-port-forward...
And realistically $600 is a pittance compared to the long-term cost of maintaining a feature that has the potential to dramatically increase the size of the code base.
Emacs will transparently handle SSH connections and let you edit remote files as if they are local. Just as importantly, if you run commands like M-x compile or M-! (aka M-x shell-command) while you're looking at a remote file, Emacs will "do the right thing" and run the process on the remote machine, rather than on your local machine.
This gives you the benefits that the author mentions about mosh - characters will appear as you type without waiting on a server round-trip. Additionally, you also won't have to wait for a server round-trip to see your cursor jump to the minibuffer, which the author mentions as a pain point.
If you need to run things at the command-line, you can do so in a "shell" buffer via M-x shell (or M-x term if you need to run something curses-like).
The major downsides:
- I/O operations (including opening and saving files) can be much slower than if you were running Emacs in a terminal, because the bytes need to be sent over SSH.
- It's not always easy to keep track of which buffers are "remote" and which are "local", so sometimes you'll run a command and it will run a process on a different machine than you expected.
> This is accomplished using a new protocol called the State Synchronization Protocol, for which Mosh is the first application.
This protocol has never been documented, making it "only" rather than "first", and once again putting the world in the position of one particular implementation's foibles being the only "specification".
It's easier for me to just pick one terminal emulator I like and spawn multiple windows of it, ssh about and deal with the woes of TCP than put up with VT layering like mosh requires.
I really like that I can close my laptop and put it into standby, take it somewhere else with a different wifi network and just open it and get right back to work. From a user perspective it appears as if mosh was never disconnected.
mosh is for low bandwidth high latency but as a side effect it makes the reconnect step transparent most of the time.
(for the record i also use both mosh and tmux)
However, I quickly ran into a problem which is that it breaks a scrollback. This is the one big problem with mosh and will probably never be fixed. I tried to find other ways but eventually decided I couldn't live with it. Shame.
Anyway that means if you cat some long file and then hit control-c, whatever was output before you aborted still has to be sent to your terminal emulator so it can write it to this paper (save it to the scrollback buffer). This sucks on a slow connection. mosh is running the emulator on the remote side of the connection and the client that you're interacting with locally just syncs periodically to whatever is currently being shown on the screen. So it doesn't have to send all of the output, which as you've seen, does break local scrollback. But it's a great solution for the problem it was designed to solve which is making terminals work over slow and/or unreliable network connections.
If you pair it with gnu screen or tmux, then they become the terminal emulator and maintain the scrollback buffer, so you can still scroll back and see all of the paper. You just can't do it with the slider on your local window because your local emulator doesn't have access to the full contents of the buffer. I turn off scrollbars in rxvt and just rely on screen to maintain the buffer and use command keys to go back when I need to.
But it wouldn't be exceptionally difficult to figure out some way to stuff lines into the scroll buffer. Especially if you got a mosh-aware terminal.
alias server='mosh server -- screen -xRR -D'
If mosh could do this, I would `alias ssh=mosh` and use it all the time.
Otherwise I need to remember which servers have mosh installed, and for those that don’t I need to ssh in, apt-get mosh, disconnect, and reconnect with mosh.
I’d imagine it could either just scp a binary over to the target system (in the case of insufficient privileges) or use the system package manager to install itself.
iTerm2’s tmux integration is pretty nice, though.
Apparently there is iTerm2+tmux+mosh dark magic if one is into that kind of thing: https://www.gaeblogx.com/2017/10/09/Remote-Shell-Session-Set...
Reminds me of pyqtgraph, which gets commits most days last time I checked yet hasn't released since 2016.
About say 10% of my reconnects would fail with mosh, and the session would be orphaned. But combining with tmux, even if I lost the mosh session I could make a new one and reattach to tmux.
My use case was iPad/iPhone remote working. (Usually while visiting in-laws or on a plane) Very intermittent connectivity.
I used the open source blink terminal for iOS (which I bought because it’s great). It can use mosh.
Since the mesh is UDP only, this provides a relatively normal shell experience for this type of situation.
Works well.
It'd be great if I could use it now while working from home because my ssh sessions keep dying. Unfortunately the work firewall won't allow this. So tmux to preserve the sessions is the best I can do.
If you spend any time using a terminal to remotely manage servers, it's worth using at least tmux (and mosh because it provides a better experience with imperfect connections).
The difficulty is key agreement and authentication, but both those happen in SSH itself. So by the time Mosh is invoked you're definitely an authorised user and you and the server share a secret, so Mosh is just moving encrypted packets.
It uses authenticated encryption, and I don't know much about the specific mode it apparently uses, OCB3, but in general there's just not that much to go wrong over and above all the work still happening in SSH.