Graceful Shutdown in Go: Practical Patterns
victoriametrics.com
victoriametrics.com
Adding a global preStop hook with a 15 second sleep did wonders for our HTTP 503 rates. This creates time between when the loadbalancer deregistration gets kicked off, and when SIGTERM is actually passed to the application, which in turn simplifies a lot of the application-side handling.
IMHO, there are two things that kubernetes could improve on:
1. Pods should be removed from Endoints _before_ initiating the shutdown sequence. Like the termination grace, there should be an option for termination delay. 2. PDB should allow an option for recreation _before_ eviction.
it's also possible, if you're not careful, to lose the last few seconds of logs from when your service is shutting down. for example, if you write to a log file that is watched by a sidecar process such as Promtail or Vector, and on startup the service truncates and starts writing to that same path, you've got a race condition that can cause you to lose logs from the shutdown.
Then after you go through all that effort most of the data is utterly ignored and rarely are the business insights much better then the trailer park version ssh'ing into a box and greping a log file to find the error output.
Like we put so much effort into this ecosystem but I don't think it has paid us back with any significant increase in uptime, performance, or ergonomics.
This dilutes the good and helpful stuff with marketing bullshit.
Grafana seemingly inventing new time series databases and engines every few months is absolutely painful to try keep up to date with in order to make informed decisions.
So much so I've started using rrdtool/smokeping again.
but many systems are more complicated than that. the observability ecosystem exists for a reason, there is a real problem that it's solving.
for example, your app might outgrow running on a single box. now you need to SSH into N different hosts and grep the log file from all of them. or you invent your own version of log-shipping with a shell script that does SCP in a loop.
going a step further, you might put those boxes into an auto-scaling group so that they would scale up and down automatically based on demand. now you really want some form of automatic log-shipping, or every time a host in the ASG gets terminated, you're throwing away the logs of whatever traffic it served during its lifetime.
or, maybe you notice a performance regression and narrow it down to one particular API endpoint being slow. often it's helpful to be able to graph the response duration of that endpoint over time. has it been slowing down gradually, or did the response time increase suddenly? if it was a sudden increase, what else happened around the same time? maybe a code deployment, maybe a database configuration change, etc.
perhaps the service you operate isn't standalone, but instead interacts with services written by other teams at your company. when something goes wrong with the system as a whole, how do you go about root-causing the problem? how do you trace the lifecycle of a request or operation through all those different systems?
when something goes wrong, you SSH to the box and look at the log file...but how do you know something went wrong to begin with? do you rely solely on user complaints hitting your support@ email? or do you have monitoring rules that will proactively notify you if a "huh, that should never happen" thing is happening?
If the log field doesn't change with every log entry, then good databases for logs (such as VictoriaLogs) compress such a field by 1000x and more times, so its' storage space usage can be ignored, and it doesn't affect query performance in any way.
Storing many fields per every log entry simplifies further analysis of these logs, since you can get all the needed information from a single log entry instead of jumping over big number of interconnected logs. This also improves analysis of logs at scale by filtering and grouping the logs by any subset of numerous fields. Such logs with big number of fields are named "wide events". See the following excellent article about this type of logs - https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide... .
Coroot is an open source project I'm working with to try and to tackle this. eBPF automatically gathers your data into a centralized service map, and then the tool provides RCA insights (with things like mapped incident timeframes) to help implement fixes quicker and improve uptime.
GitHub here and we'd love any feedback if you think it can help: https://github.com/coroot/coroot
It (will) cover the stuff like "sync the logs"/"wait for ingresses to catch up with the liveness handler"/etc.
https://github.com/utrack/caisson-go/blob/main/caiapp/caiapp...
https://github.com/utrack/caisson-go/tree/main/closer
The docs are sparse and some things aren't covered yet; however I'm planning to do the first release once I'm back from a holiday.
In the end, this will be a meta-platform (carefully crafted building blocks), and a reference platform library, covering a typical k8s/otel/grpc+http infrastructure.
however, there are some benefits to the pull model, which is why I think Prometheus does it by default.
with a push model, your service needs to spawn a background thread/goroutine/whatever that pushes metrics on a given interval.
if that background thread crashes or hangs, metrics from that service instance stop getting reported. how do you detect that, and fire an alert about it happening?
"cloud-native" gets thrown around as a buzzword, but this is an example where it's actually meaningful. Prometheus assumes that whatever service you're trying to monitor, you're probably already registering each instance in a service-discovery system of some kind, so that other things (such as a load-balancer) know where to find it.
you tell Prometheus how to query that service-discovery system (Kubernetes, for example [1]) and it will automatically discover all your service instances, and start scraping their /metrics endpoints.
this provides an elegant solution to the "how do you monitor a service that is up and running, except its metrics-reporting thread has crashed?" problem. if it's up and running, it should be registered for service-discovery, and Prometheus can trivially record (this is the `up` metric) if it discovers a service but it's not responding to /metrics requests.
and this greatly simplifies the client-side metrics implementation, because you don't need a separate metrics thread in your service. you don't need to ensure it runs forever and never hangs and always retries and all that. you just need to implement a single HTTP GET endpoint, and have it return text in a format simple enough that you can sprintf it yourself if you need to.
for a more theoretical understanding, you can also look at it in terms of the "supervision trees" popularized by Erlang. parents monitor their children, by pulling status from them. children are not responsible for pushing status reports to their parents (or siblings). with the push model, you have a supervision graph instead of a supervision tree, with all the added complexity that entails.
0: https://prometheus.io/docs/instrumenting/pushing/
1: https://prometheus.io/docs/prometheus/latest/configuration/c...
Have you come across any convenient solution for this? If my scrape interval is 15 seconds, I don't exactly have 30 seconds to record two scrapes.
This behavior has sort of been the reason why our services still use statsd since the push-based model doesn't see this problem.
package main
import (
"fmt"
"log"
)
func main() {
defer fmt.Println("in defer")
log.Fatal("fatal")
}
package main
import (
"fmt"
"log"
)
func main() {
defer fmt.Println("in defer")
panic("fatal")
}
That my application went down from sig int makes a big difference compared to kill.
Blue-Green migrations for example require a graceful exit behavior.
it may not always be necessary. e.g. if you are deploying a new version of a stateless backend service, and there is a load balancer forwarding traffic to current version and new version backends, the load balancer could be responsible for cutting over, allowing in flight requests to be processed by the current version backends while only forwarding new requests to the new backends. then the old backends could be ungracefully terminated once the LB says they are not processing any requests.
The only way to make sure a system can handle components hard crashing, is if hard crashing is a normal thing that happens all the time.
All glory to the chaos monkey!
In a second thought though, maybe I do. That might be the only way to ensure the assumption is true. Like the Netflix's chaos monkey thing a couple years ago.
That was released 15 years ago.
It is relatively straightforward to implement under systemd. And nginx has been supporting that for over 20 years. Sadly Kuberenets and Docker have no support for that assuming it is done in load balancer or the reverse proxy.
Sometimes it's not enough and you have to 'do it by hand', but generally if you're working in a system that has GC, freeing memory is not something that you think of often.
The BEAM is designed for building distributed, fault tolerant systems in the sense that these type of concerns are first class objects, as compared to having them as external libraries (eg. Kafka) or completely outside of the system (eg. Kubernetes).
The three points the author lists in the beginning of the article are already built in and their behavior are described rather than implemented, which is what I think OP meant with not having to 'intentionally create graceful shutdown routines'.
- Some applications want to instantly terminate upon receiving kill sigs, others want to handle them, OP shows how to handle them
- In the case of HTTP servers, you want to stop listening for new requests, but finish handling current ones under a timer. TBF, OPs post actually handles that badly with a time.Sleep when there's a running connection, instead of using a sync.WaitGroup like most applications would want to do
- Regardless if the application is GCd or not, you probably want to still manually close connections, so you can handle any possible errors (a lot of connections stuff flushes data on close)
Blog post author has to hand roll these, including picking the wrong solution with time.Sleep as you mentioned.
My analogy with GC was in that spirit; if GC is built in, you don't need custom allocators, memory debuggers etc 99% of the time because you won't be poking around memory the same way that you would in say C. Malloc/free still happens.
Likewise, graceful shutdown, trapping signals, restarting queues, managing restart strategies for subsystems, service monitoring, timeouts, retries, fault recovery, caching, system wide (as in distributed) error handling, system wide debugging, system wide tracing... and so on, are already there on the BEAM.
This is not the case for other runtimes. Instead, to the extent that you can achieve these functionalities from within your runtime at all (without relying on completely external software like Kubernetes, Redis, Datadog etc), you do so by glueing together a tonne of libraries that might or might not gel nicely.
The BEAM is built specifically for the domain "send many small but important messages across the world without falling over", and it shows. They've been incrementally improving it for some ~35 years, there's very few known unknowns left.
I find that I typically have a few services that I need to start-up and sometimes they have different mechanisms for start-up and shutdown. Sometimes you need to instantiate an object first, sometimes you have a context you want to cancel, other times you have a "Stop" method to call.
I designed the library to help my consolidate this all in one place with a unified API.
https://pkg.go.dev/git.sr.ht/~mariusor/wrapper#example-Regis...
> The exact wait time depends on your readiness probe configuration
A terminating pod is not ready by definition. The service will also mark the endpoint as terminating (and as not ready). This occurs on the transition into Terminating; you don't have to fail a readiness check to cause it.
(I don't know about the ordering of the SIGTERM & the various updates to the objects such as Pod.status or the endpoint slice; there might be a small window after SIGTERM where you could still get a connection, but it isn't the large "until we fail a readiness check" TFA implies.)
(And as someone who manages clusters, honestly that infintesimal window probably doesn't matter. Just stop accepting new connections, gracefully close existing ones, and terminate reasonably fast. But I feel like half of the apps I work with fall into either a bucket of "handle SIGTERM & take forever to terminate" or "fail to handle SIGTERM (and take forever to terminate)".
Then the main app goroutine can close the context when it wants to shutdown, and block on the waitgroup until everything is closed.
So if you send a SIG-INT/-TERM signal to the server there's a delay to clean up resources, during which the new requests get served a response that doesn't try to access them and fail in unexpected ways, but a configurable "not in service" error.