Hacker News Reader

Comment by jpollock | Hacker News Reader

Process upgrades are a variant of fail-over on either hardware death or bug. I recommend treating upgrades as a chance to test your failure recovery processes.

If you really can't afford someone getting a "connection refused" what happens when the machine's network connection dies?

View on HN

You're right, but in some cases, for example a 'git push' to deploy an application, we don't want the SSH connection to be cut when upgrading our SSH server, this is not an option. If the server crashes the current connections will be lost, but it should be the only scenario when this occures.

You can cut an SSH connection while SSHed in providing the server comes back up in a timely manner as SSH does some clever keep alive tricks to resume broken connections. I often restarted sshd or even the network interfaces themselves while SSHed into some Linux boxes.

That's because when you connect to your SSH, it forked and there is one process which is handling your connection, then when you restart the daemon, it only restarts the master, no harm done to the existing connections.

That doesn't help when you restart the network interfaces though. Normal TCP/IP behaviour would be for the connection to terminate, however SSH can reattach itself.

I don't want to say anything wrong on this and I don't have the knowledge on this, but it's possible than the OS is doing a lot of work on this to avoid cutting connections. Do you have any details on this?

Sure, not a problem:

"A keep-alive is a small piece of data transmitted between a client and a server to ensure that the connection is still open or to keep the connection open. Many protocols implement this as a way of cleaning up dead connections to the server. If a client does not respond, the connection is closed.

SSH does not enable this by default. There are pros and cons to this. A major pro is that under a lot of conditions if you disconnect from the Internet, your connection will be usable when you reconnect. For those who drop out of WiFi a lot, this is a major plus when you discover you don't need to login again."

Source: http://www.symkat.com/ssh-tips-and-tricks-you-need

There's probably better sources out there, that was just one of the top results in Google as, if I'm honest, I'm not an expert on this either.

Devil is in the details, but the source actually talks against you here:

  This happens because your router or firewall is trying to clean up dead connections. It's seeing that no data has been  transmitted in N seconds and falsely assumes that the  connection is no longer in use.
 To rectify this you can add a Keep-Alive. This will ensure that your connection stays open to the server and the firewall doesn't close it.

In other words: What keep-alive does is that it prevents routers/middle-ware-boxes to forget that the connection exists in the first place. This is not needed on a clean internet connection where everything is treated as stateless and simple routing is everything that is done.

It also says: "if you disconnect from the Internet, your connection will be usable when you reconnect.". The same logic applies if you restart your network interfaces on the server.

I'm open to being proved wrong here, but as I've already said, only been doing this for several years now, so I'd need a counter argument to explain the mechanics of what's allowing the connection to reattach rather than "it's not possible" :)

edit: hmmm, re-reading the latter part of keep alive article I posted, it does seem to imply what your saying. So how come my SSH connections aren't nuked then? Is this just a property of TCP/IP (I'm not a networking guy so ignorant to some of the lower level stuff)

Yes, as long as both endpoints (TCP/IP stacks) keep their state it doesn't matter what stuff does between them. Interfaces being one of the things being between stacks. That is what allows for stuff like live application migration as long as you bring the TCP/IP state.

I see. Thank you :)

I'm pretty sure you're wrong. Mind citing a source on that the connection should terminate?

I'm fairly certain that the connection stays alive and that SSHd doesn't need to care about this, all handled on OS-level. That's why if you change IP or something like that, it doesn't "reattach" as you call it.

You might be right, however I wouldn't expect an IP change to reattach since that would be a massive security vulnerability.

For us it's about not dropping your customers requests when we can, obviously that happens sometimes, but there's no need to do that if you can avoid it during normal deployment cycles.

Reply on news.ycombinator.com