Hacker News Reader

Comment by danwee | Hacker News Reader

How does one handle zero downtime deployments with single-file golang binaries? I remember I tried this setup some time ago and I couldn't successfully manage cleanly to accomplish no downtime when deploying a new version of my service. The reason was mainly port reuse. I couldn't have the old and the new version of my service running on the same port... so I started to hack together something and it became dirty pretty quickly. I'm talking about deployment of new version of service on the same machine/server as the old version was running.

View on HN

Some of this is solved by using e.g. systemd, depending on your needs.

> I couldn't have the old and the new version of my service running on the same port...

You can, actually! You just can’t open the port twice by default. So one or both of the processes needs to inherit the port from a parent process, get passed the port over a socket (Unix sockets can transmit file descriptors), or use SO_REUSEADDR.

There are some libraries that abstract this, and some of this is provided by tools like systemd.

Some of this is probably going to have to be done in your application—like, once your new version starts, the old version should stop accepting new connections and finish the requests it has already started.

So can I just start another process with SO_REUSEADDR and gracefully shutdown the old process?

The master/worker thing that nginx / gunicorn et al do is pretty neat, but relies on signals; so seems pretty messy and error prone to write yourself.

You may have to start both processes with SO_REUSEADDR, I don’t remember the exact semantics.

People have a healthy skepticism of signals from the C days, but if we’re talking about Go, you’d just call signal.Notify. Any way of signaling your app to shut down works, though.

https://pkg.go.dev/os/signal@go1.20.2#Notify

> Some of this is probably going to have to be done in your application...

FWICT tableflip does exactly this: https://github.com/cloudflare/tableflip

Same way you do with any other app not specifically designed for it; you start 2 copies of it and put loadbalancer in front of it. I did that via some systemd voodoo

But TECHNICALLY to do that in one without external proxy you'd need to figure out how to set SO_REUSEPORT for the web socket handler, then start the second one before the first.

Haven't actually tried it but someone apparently did: https://iximiuz.com/en/posts/go-net-http-setsockopt-example/

You'd still have any ongoing connections cut unless you unbind socket and then finish any existing connection, which would be pretty hard with default http server.

I just put HAProxy instance on my VPS that does all of that, including only allowing traffic once app says "yes I am ok" in healthcheck. Then the app can have "shutting down" phase, where it reports "I am down" on healthcheck but still finished any active connections to the client.

This doesn’t sound Go-specific, if you use something like haproxy targeting multiple nodes you can take them down one by one to perform a rolling upgrade.

Socket activation via systemd[0] is an option, assuming you are fine with certain requests taking a longer time to complete (if they arrive while the service is being restarted). Otherwise using a proxy in front of your app is your best bet (which has other benefits too, as you can offload TLS and request logging/instrumentation).

- https://github.com/bojanz/httpx#systemd-setup

If you really don't want to use different ports you can handle it with Docker. Since each container has its own IP, they can all expose the same port. Otherwise, for non-containerized deployments you'll have to resort to two different ports.

In either case, you will need a reverse proxy like Traefik/Nginx in front to smartly "balance" incoming requests to the two instances of the service.

I guess this is a problem inherent not just in a single-file go app, but in any deployment where the whole stack is contained within a single process.

The post says the process starts up quick enough that the process being temporarily unavailable isn't noticeable - but what if the process _doesn't come back_? It's also impossible to do blue/green deployments this way.

It's clearly not a solution suitable to large-scale deployments. The simplicity has its trade-offs.

If you want to do deployments with single-file apps or other "whole stack in a single process" type of deployments there are other options to do it with zero-downtime.

One good option would be to spin up a second server/instance/container, run binary on new system, ensure it's good, once comfortable then swap DNS entry to the new system.

Can't help with how to implement this, but just to be sure: You should be able to use the same port in multiple instances if you bind those with SO_REUSEPORT. A quick search points to https://github.com/libp2p/go-reuseport for an implementation. Now you just need a mechanism to drain the old process.

Rough psuedocode to do this with the built-in http.Server where startServer(..) would use the reuseport library to create the listener so multiple servers can listen within the same process:

  func reloadConfig(config) {
    if newServer, err := startServer(config); err != nil {
      // gracefully shutdown previous server
      // no new connections will go to old server
      oldServer.Shutdown(...)
      oldServer = newServer
    }
  }

You don't. You can sort of emulate it with services that stateless, load-balanced, and L7 proxied.

If you want stateful zero downtime deployments, use Elixir or Erlang that has the ability to live migrate data from one version of code to the next.

You can always share ports.

But the one way to do no downtime deployments is to have more than one server.

I've always run services behind a proxy. Spin up a new server with the code (works for any type of deployment). Validate it's up. Switch the proxy from the old to new server.

This is where the simplicity of single-file golang deployments falls short.

Just make sure you’re not slowly recreating bad, homebrew versions of all of the nice things that Kubernetes does in an attempt to turn a simple deployment into a production ready deployment.

Reply on news.ycombinator.com