Hacker News Reader

Comment by bragr | Hacker News Reader

I've implemented this a few times in a few languages based on exactly what nginx does. It works well, and it is pretty straight forward if you are comfortable with posix style signals, sockets, and daemons.

I'm not sure it is super critical in the age of containerized workloads with rolling deploys but at the very least the connection draining is a good pattern to implement to prevent deploy/scaling related error spikes.

View on HN

Even with containerized workloads, you still have an ingress, or SPOF (or multiple, when using multicast), and the seamless restart is meant for exactly those processes. Nginx is often used (https://kubernetes.github.io/ingress-nginx/), or when you use AWS, GCS etc they provide such a service for you.

Not sure how the cloud providers do it though, maybe combination of low DNS TTL and rolling restart since they often have huge fleets of servers which handle ingress?

A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers.

With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.

I think the parent was talking more about the fact that at some point, you have a component that should be available as much as possible. In the case you mention, that would be the load balancer. Being able to upgrade it in place might be easier than other ways.

How do you restart the load balancer though, without dropping traffic?

You would need more than one to do a rolling restart. Alternatively to do it with one instance of a software load balancer is a bit more work, spin another instance up and update DNS. Wait for traffic to the old one to die as TTLs expire, then decommission.

But I agree it isn't as easy as a in place upgrade.

2 LBs and a VIP/DNS switch.

two nginx load balancers, reroute to the secondary via dns, restart primary

If you really want to have no SPOF you'd probably build something like this:

Multihomed IP <-> Loadbalancer <-> Application

By having the same setup running on multiple locations you can replace the load banacers by taking one location offline (stop announcing the corresponding route). Application instances can be replaced by taking the application instance out of the load balancer.

> Not sure how the cloud providers do it though

At least some I am familiar with operate at the packet level and can hand off live "connections" to a peer or hot standby, along with full session state.

Remember that with TCP or anything else, the abstract session is an illusion.

Reply on news.ycombinator.com