Gracefully Restarting a Go Program Without Downtime | Hacker News Reader

Gracefully Restarting a Go Program Without Downtime

gravitational.com

122 pointsaberoham45 comments

This kind of thing is foreign to me because I can't imagine writing an application, to run in the cloud (as I always do) where I care if a single instance has some downtime. I always rely on something at a higher level proxying my traffic to the application, and when I need to push my application I just shoot each instance in the face and start another one elsewhere, usually on a different host:port. When I'm using the cheapest cloud capacity I can buy my application is constantly preempted by higher-paying tenants anyway, so I have to write my systems as if any instance will suddenly stop. Once you have that problem you can forget about graceful in-place restarts forever.

> Once you have that problem you can forget about graceful in-place restarts forever.

Sure, unless you're the one running the load balancer or the service discovery system that enables other people to just kill their instances without worrying about graceful restarts.

See my other comment on this thread. Even in that case you don't have to care.

There are tons of applications where rolling upgrade doesn't necessarily work out.. anything that is even remotely latency sensitive or state-heavy, for example a game server or an order matching engine

Every service that Google operates is released with rolling updates and is fully preemptible, including the most highly latency-sensitive services and all of the stateful storage services. Google is an existence proof for the applicability of rolling updates to services of all kinds.

Not every service

For state heavy applications, you want graceful restart even if you can tolerate hard failures.

I would say this approach is for situation where the model you're using isn't always applicable.

In the case of this article, it's discussing teleport, a drop in replacement SSH server. For many users, this is their administrative connection to a fleet of servers (teleport can operate as a transparent bastion, as well as on the end host). So in our case, it's a desirable feature to not interrupt existing shell sessions, while replacing the software serving those connections. This could mean the admin's own connection to the server doesn't fail mid-upgrade, or that hundreds of internal employee's don't groan because whatever they were doing that used a shell just got dropped and they need to restart.

Disclaimer: I work with the author of the original article, and contributed to the article.

I don't take any issue with treating this as a special case scenario, although I think the modern trend is to treat the need for long-lived SSH sessions for managing services as a design smell.

What I do take issue with is the suggestion that handling such special cases is "table stakes." A little humility goes a long way in bolstering credibility.

I agree. The characterization of live reload capability as "table stakes" for availability is nonsense. High-availability designs never require singleton services.

Being able to replace vulnerable singletons that require process and TCP socket gymnastics with multiple instances having good fault isolation properties is the real "table stakes" for modern highly-available service architectures.

You know that some people work on servers that are stateful? Not everyone use REST type of transport where a failed node is ok because the lb will route traffic to another place. Your solution doesn't work for things like game servers.

Presumably you just drain the servers running the old version as games end and migrate traffic to the servers running the new version?

Even stateful services like game servers could be organised to handle losing a node. It's harder to do and you need some state replication in place. (versioned state if you want to do upgrades that way) It may not be cost effective to do so. But the possibility is there.

I worked on a plan to have live migration of multiparty phone calls between data centres in realtime. The tech is available. But your planning outcome may be: it's cheaper to make single node more stable and deal with the occasional failure.

"Stateful" only means you need to care about the state being moved. Things like Java hotswap, Erlang live code reload, and your custom solutions can handle that.

Do you run game servers? They absolutely have failover capability.

My usecase is that I have 1500+ raspberry pi class machines running my daemon process (golang agent). I want to be able to upgrade the binary, without downtime.

Next time consider using Elixir Nerves. This capability is built in to the deployment framework.

Thanks for the suggestion. It seems like Nerves needs a whole firmware to support it. These are machines where the most I can do is copy a binary onto them (crypto mining hardware). golang was the simplest solution for cross compiling I could find at the time.

I can update Erlang programs _while they're running_ without any downtime. We do this often at our company.

See http://erlang.org/doc/design_principles/release_handling.htm... for the basics. With no shared state, and no global variables, and no side effects, you can update functions so the next time they're called they're something else.

Erlang is badass in this way

What a coincidence seeing this topic today. Just yesterday I released some similar code for upgrading Go servers in-place while maintaining sockets:

https://github.com/crawshaw/littleboss

It is possible to pass file descriptors using os/exec, using the ExtraFiles attribute [1]. No need drop down to os.StartProcess.

1: https://golang.org/pkg/os/exec/#Cmd

Fantastic examples. I appreciate that they allude to the (canonical?) example of how Nginx does zero downtime upgrades using SO_REUSEPORT. I had hard of the trick of sharing the FD with a forked process but it's nice to a working example.

It's theoretically interesting to hand off a listen socket, but I'd rather take 1/n machines out of service than leave half of my memory unused on every machine.

Edit: I also remember hearing some out-there stuff about handing off your IP address and open connection state to another machine so the client doesn't even notice…

Will Tarreau of haproxy has written an insightful post on the same topic:

"Truly Seamless Reloads with HAProxy – No More Hacks!"

https://www.haproxy.com/blog/truly-seamless-reloads-with-hap...

I like to use a simple protocol: 1) Server receives shutdown signal 2) Server tells all clients to disconnect, and also tells the load balancer to not send more traffic 3) Client's connect to another server or waits for the server to restart 4) server exit

How do you tell the load balancer to stop sending traffic? I've done two ways in previous jobs:

- Refuse all new connections. If you can change the program, this can be done by closing the listening socket (I think) , but I just installed an iptables rule.

- use the load balancer's API to offline the server(at the time it was F5).

I liked the first solution because it was load balancer agnostic, but required retries by the load balancer or client to avoid disruptions. Curious to hear what other people did.

See the lameduck section of this chapter of the Google SRE book. They do frequent health checks on all running jobs. If a health check fails they stop sending that job traffic. The concept of lameduck is you report yourself as unhealthy but are still available to serve traffic, you wait before all of your clients that were loaded balanced to you before you stop a sending traffic.

https://landing.google.com/sre/book/chapters/load-balancing-...

Health checks are part of GCPs load balancer. https://cloud.google.com/compute/docs/load-balancing/health-...

I think it's best if the client can handle the load balancing by connecting to a random server, and have some servers built into the source for bootstrapping. The client can know what to do if the connection closes and handle interruptions.

This is an excellent write up, very easy to follow. Overseer [1] is a pretty good implementation as well.

"Commonly, graceful restarts are performed by the active process (dark blue) closing its listeners and passing these matching listening socket files (green) over to a newly started process. This restart causes any foreground process monitoring to incorrectly detect a program crash. overseer attempts to solve this by using a small process to perform this socket file exchange and proxying signals and exit code from the active process."

[1] https://github.com/jpillora/overseer

I'm using socketmaster[1] to do the graceful update or just normal service reload. It is basically another process to facilitate socket exchange between 2 process. Just make sure that the service able to shutdown gracefully.

However this approach need the binary to be started from socketmaster (as child process). I'm not sure what is the real cons tho, except the fact that it alter SIGHUP behavior - which is fine in this case imo.

[1] https://github.com/zimbatm/socketmaster

https://github.com/paytm/grace The graceful handle on the service side can use this package

Another reference project: https://github.com/facebookgo/grace

Seems simpler to do this at the load balancer level. That way you can use the same strategy for any type of application.

What if your Go application IS a load balancer?

Then you should use nginx or haproxy ;)

Even in that case you don't need graceful in-place restarts. If you have multiple replicas of the load balancer and ECMP from higher on your stack, all you need is for your balancer process to 1) remove itself as a destination for new flows; 2) stop accepting new connections and complete all existing connections; 3) shut down and restart; 4) add itself back to the set of new flow destinations.

If you don't have multiple instances of your load balancer, that's weird.

I don't believe this approach would work for a layer 7 load balancer (where the connection is terminated on the host). This is in regards to the original article, which is discussing how to maintain socket state through a process upgrade.

When you remove the host from the routing table, the ECMP algorithm will redistribute the traffic to the other hosts still in the pool. For TCP connections, this will generate reset's, and drop any active connections moved between hosts. Also, at times I've not gotten clear answers from router vendors on whether they support a consistent hash in their ECMP implementation, which means that connections through the load balancers not being reloaded may also be rebalanced to other hosts and be affected by the routing table change.

Even if we consider going down the stack to a layer 4 load balancer, outside the scope of this article, it may still be desirable to combine approaches and offer hitless reloads (a common case) and not just recover from a node failure (a less common case).

The reason is, that you would need state replication of the load balancing tables to support removing the load balancer from the routing tables, and having another load balancer take over it's work while maintaining the connections through the load balancer. This is under the assumption that the service being load balanced requires an uninterrupted connection of course.

Disclaimer: I work with the author of the article and contributed to it.

I've worked with ECMP routers where you can inform that router you'd like new flows to be sent elsewhere while existing flows continue to be sent to their established endpoints. JunOS for example has this feature.

Do you recall what option/platform that is? I assume that must have been something like an SRX, not an MX/QFX/EX.

I made a blog post a couple years ago about doing this for a mud in Go using the FD passing approach.

http://austinhanson.com/slinging-mud-graceful-reboots

Does anyone have any good articles about the same technique for Node?

“modern systems software”

Lol, no. Modern, professional software development uses a load balancer to manage a canary/red-black deployment strategy. Zero downtime deployments to a live server, outside of telecom/finance, is a sign of an amateurish team.

Reply on news.ycombinator.com