Eternal Terminal: Remote terminal for the busy and impatient
mistertea.github.io
mistertea.github.io
ET however uses TCP, and some type of BackedReader and BackedWriter for TCP data - which _implies_ to me that everything is being sent/received in a massive queue. Also, it supports scrolling, which would be difficult _(but not impossible)_ if they only cared about recent frames like Mosh.
Finally, I'm curious how well it handles mobile connections. That's another selling point of Mosh. Admittedly I never use mobile internet, so that's not a concern of mine, personally.
I love Mosh and I'm always happy to see competing apps, hope they complement each other well :)
Presumably with the TCP URG flag. That said, I only read years ago that's how it's done, without ever checking myself. I've found references to in in telnet, but not ssh, so perhaps the latter uses some other mechanism.
The way you'd want to use it is not to transmit the Ctrl-C urgently (that direction is clear), but to transmit the interruption of the reply urgently back to the client. And I think there's nothing that will save the server from having to actually send all the bytes that you managed to attempt to send before you received the Ctrl-C.
Also, it's completely broken: https://tools.ietf.org/html/rfc6093
mosh works around this by rendering the terminal window server-side and sending diffs of the terminal window (think VNC or streaming video) to the client. So the in-flight data from server to client is at most a screenful. If the data isn't sent in the first place, you don't have to worry about unsending it. (Of course the tradeoff is loss of scrollback.)
That could always be buffered on the server side.
You can probably do some things to heuristically improve scrollback, e.g., telling the terminal sync thing to sync 1000 lines of scrollback if the connection seems good and only the current screen if it seems poor. You won't get reliable scrollback but maybe it's good enough.
Both scrollback and tmux -CC support are feasible in theory with mosh, though difficult to get right:
https://news.ycombinator.com/item?id=14228591#14229268
https://www.mail-archive.com/mosh-devel@mit.edu/msg00304.htm...
There's a fork of mosh linked in the first comment that switches the terminal-to-user end from a synchronized terminal screen to a bytestream (same as the user-to-terminal direction, i.e., keyboard input). It's not going to be robust to, say, catting a huge file and needing to Ctrl-C, but it should maintain mosh's robustness to radio (wifi/cell) network blips. It was designed for scrollback, but I'd sort of expect it to make tmux -CC work too.
Anyway, it is super cool to see more efforts in this space, and I bet for people who work primarily at desks with reliable internet connections, this is a huge improvement over, say, autossh. But for my own use case for mosh, this doesn't seem like it's got feature-parity yet. Maybe there are clever things you can do with aggressively restarting TCP connections that get you the same experience as mosh's UDP approach.
Is this the "Scroll alternate screen" checkbox? I believe that just controls the scroll wheel, and makes it not do native scrolling and instead just send a page-up or something. If so, then non-native scrolling is definitely possible ... but it's already possible if you run screen or tmux instead of mosh. (Which is what I do, and it works fine for me, in part because I learned how to use screen's scrollback even before mosh existed.) I can see the argument for exposing this with better keyboard shortcuts than screen's inscrutable ones, but my impression is that people mostly want native scrolling so they can, e.g., drag in the terminal UI to copy and paste a large amount of text. You can't do that with non-native scrollback, since the text isn't all contiguous in the terminal window.
That said, it's not as friendly to IP changes, nor does it provide predictive typing, so there's definitely something to be said for the value of tools such as these.
I do have one question, given the tmux and scrolling support, how much mosh-like lag compensation can ET provide?
The client will time out the server by sending a packet with no response (such as via the 'ServerAliveInterval'), the server via unsetting 'ClientAliveInterval', 'ClientAliveCountMax', and 'TCPKeepAlive'.
You can break the connection in between and resume as long as your ip hasn't changed.
So it's not really resuming ssh or anything, just a function of tcp sockets. Generally disabled on public since something could open up a bunch of listening sockets and then leave them and never close.
With your servers that you've disabled the keep alive and intervals, if you drop a connection and never reconnect, it's going to keep it open. A malicious could keep opening connections and exhausting your server.
The Linux kernel will close the TCP connection if it's idle after about 5 minutes (configurable), regardless of the application settings (with one exception noted in a bit). That's why you can even get "connection closed by peer" errors in the first place. TCP keepalive messages are used to circumvent the kernel (and every router between the two endpoints) from closing that TCP connection due to it being idle.
Plus, I can put my computer into hibernation for upwards of 12 hours, and still come in and have a responsive SSH session over a new TCP connection. So long as the key being used to encrypt the session is still in use, and the associated TTY has not been closed out, your client can create a new TCP connection and resume the previous SSH session.
> A malicious could keep opening connections and exhausting your server.
This could (and does) occur regardless of your application TCP settings.
TCP keepalive messages are what the kernel uses, and not what circumvents the kernel's own probes.
Applications can also implementation their own layers of 'are you alive?'/'yes I am' type messages on top. SSH has its own messages (see the ServerAliveInterval/ServerAliveCountMax config keys)
What's happening is not that your client is creating a new TCP connection and resumes an SSH session, you are just and continuing on your existing SSH session over an existing TCP connection - which you can do as long as neither the client ssh process or server sshd process have been restarted, and neither end has attempted and failed to send any data or otherwise timed/errored out the TCP connection.
(e.g. if you did a (sleep 60 ; echo test)& , then went hibernate on your client side, the server side would attempt to send some output in 60 seconds, and eventually time out the connection after a few minutes unless you wake up in the mean time)
Thanks for the persistence!
That said, though, there is something which closes idle, established, TCP connections, though it's not the kernel, it's iptables which does it. The timeout is something along the lines of 5 days by default, however. That's what I get for working from memory.
If you use iptables (or rather netfilter)'s stateful filter, then netfilter will forget the connection tracking state of the connection after a long-ish time without any traffic (may be 5 days, I don't remember). That doesn't close anything, it simply means that netfilter doesn't know about the connection anymore. No packets are being sent when that happens.
That in turn has two effects:
First, if you have a rule that only allows packets through for established connections, that rule will now not match packets from that connection anymore. But if the packet matches another rule that allows it to pass, then it will still make it through the firewall, and as a side effect that would then re-establish the connection tracking state, which in turn would also make the rule for established connections match again for any packets that follow.
So, in most cases, that's not in any way a problem: The next packet your client sends after a long idle time will just re-establish the connection tracking state and the connection will continue to work just fine.
Second, if the connection is NATted, the NAT state is part of the connection tracking state, and as such, the NAT mapping of the connection is also lost when the state is dropped. That means that the next packet (if it comes from the right direction) will once again consult the nat table, just as the very first packet of the connection, to establish a new NAT mapping. Now, in many cases, that will establish a new mapping that's exactly the same as the old mapping, in which case the connection also will continue to work just fine. Only if the new NAT mapping differs, that will cause the connection to fail. Yet another reason why you don't want to have NAT (and by extension, why you want IPv6).
Also, netfilter very much is in the kernel.
ClientAliveCountMax default is 3 ClientAliveInterval default is 0
Ergo by default ClientAlive messaging isn't used. If Interval were set to 60, the connection would be terminated after 3 minutes.
From man sshd_config: TCPKeepAlive "Specifies whether the system should send TCP keepalive messages to the other side. If they are sent, death of the connection or crash of one of the machines will be properly noticed. However, this means that connections will die if the route is down temporarily, and some people find it annoying. On the other hand, if TCP keepalives are not sent, sessions may hang indefinitely on the server, leaving "ghost" users and consuming server resources.
The default is yes (to send TCP keepalive messages), and the server will notice if the network goes down or the client host crashes. This avoids infinitely hanging sessions."
That's probably not that interesting because it's considered spoofable. Maybe more interesting are the client side ssh_config options which allow for opportunistic multiplexing:
ControlMaster ControlPath ControlPersist
ET is primarily made for IP changes and also because the kernel/router will sometimes kill stale connections, and it's not possible to send keep-alives if your laptop is asleep.
There is no lag compensation because ET is not a terminal emulator, it's only a remote shell. I cannot create predictive letters and undo them reliably (passing backspace commands to the terminal sometimes does not undo the last letter). Lag compensation would need to be implemented by a terminal emulator such as hyper.js.
WRT the lag compensation - this is something mosh does, so if ET does not, it's worth pointing out.
So, as it stands, persistence of the session across IP changes seems to be the primary differentiating feature. Good to know.
But mosh (and presumably this program as well) really, really shine with a laptop that you might hibernate for days, and then boot back up, and it's just as you left it.
> Mosh is a popular alternative to ET. While mosh provides the same core funtionality as ET, it does not support native scrolling nor tmux control mode (tmux -CC).
> mosh: Mosh is a popular alternative to ET. While mosh provides the same core funtionality as ET, it does not support native scrolling nor tmux control mode (tmux -CC).
Pity its not a freer license though.
https://github.com/mobile-shell/mosh/blob/master/COPYING.iOS
It's a pain to get mosh support in closed-source iOS apps, but I think that's a good thing, and that's more about the closed-source requirement. (Though the iOS requirement means that you can't take the fork-and-exec approach that e.g. JuiceSSH on Android does; your whole app has to be free software.)
There's a free-software one that charges money on the App Store for a precompiled binary, which seems like the right plan: https://github.com/blinksh/blink
I use byobu (screen) for all the things this is supposed to address. I think. Byobu doesn't keep your ssh connection from dropping but the session is still there once you reconnect with no loss of history. Plus you can connect from multiple clients (using the same user account).
That is, losing the ssh connection and needing to manually resume the session.
> ET does not implement any of the SSH protocol
That implies they rolled their own wire protocol. Maybe they did a good job (like mosh appears to have done). More likely not. I'd be OK using this on an internal network or over a VPN, but relying on it alone for security would be premature.
--Edit--
A quick glance at the repo suggests they're using NaCl to handle the encryption + the SSH server for auth
A one-time shared passkey is sent over ssh and then this passkey is used to encrypt/decrypt the data in both directions. The server and the client have different initial nonces.
What problems are you encountering? It might be worth opening a GitHub issue.
You may be running a distro that doesn't have libsodium (such as ubuntu 14). In this case you have to grab libsodium manually and build et from source.
My big frustration with both Mosh and ET is that neither expose a data communication channel along side the PTY. I use SSH for port forwarding, Tmux paste buffer sync, file transfer, and SSH agent forwarding. Without those features, I'll always need to use SSH/Autossh along side these tools for my other SSH use cases.