Zero-Downtime Kubernetes Deployments on AWS with EKS
glasskube.dev
glasskube.dev
K8S provides two (well three, now) health checks.
How this interacts with ALB is quite important.
Liveness should always return 200 OK unless you have hit some fatal condition where your container considers itself dead and wants to be restarted.
Readiness should only return 200 OK if you are ready to serve traffic.
We configure the ALB to only point to the readiness check.
So our application lifecycle looks like this:
* Container starts
* Application loads
* Liveness begins serving 200
* Some internal health checks run and set readiness state to True
* Readiness checks now return 200
* ALB checks begin passing and so pod is added to the target group
* Pod starts getting traffic.
time passes. Eventually for some reason the pod needs to shut down.
* Kube calls the preStop hook
* PreStop sends SIGUSR1 to app and waits for N seconds.
* App handler for SIGUSR1 tells readiness hook to start failing.
* ALB health checks begin failing, and no new requests should be sent.
* ALB takes the pod out of the target group.
* PreStop hook finishes waiting and returns
* Kube sends SIGTERM
* App wraps up any remaining in-flight requests and shuts down.
This allows the app to do graceful shut down, and ensures the ALB doesn't send traffic to a pod that knows it is being shut down.
Oh, and on the Readiness check - your app can use this to (temporarily) signal that it is too busy to serve more traffic. Handy as another signal you can monitor for scaling.
e: Formatting was slightly broken.
The issue with the above setup is (maybe I'm doing it wrong?) but if a pod is removed suddenly, say if it crashes, then some portion of traffic gets errors until the ALB updates. And that can be an agonizingly long time, which seemed because it's pointed at IP addresses in the cluster and not the service. It seemed like a shortcoming of the ALB. GKE doesn't have the same behavior.
I'm not the expert but found something that worked.
I definitely think the ALB Controller should be taking a more active hand in termination of pods that are targets of an ALB.
But the ALB Controller is exhibiting the same symptom I keep running into throughout Kubernetes.
The amount of "X is a problem because the pod dies too quickly before Y has a chance to clean up/whatever, so we add a preStop sleep of 30 seconds" in the Kubernetes world is truly frustrating.
Many people don’t run the main process in the container as PID 1, so this “problem” remains.
If it’s not feasible to remove something like a shell process from being the first thing that runs, exec will allow replacing the shell process with the application process.
Maybe I am holding it wrong. I'd love not to have to do this work.
But I don't see how being PID 1 or not helps (and yes, for most workloads it is PID 1)
The ALB controller is the one that would need to deregister a target from the target group, and it won't until the pod is gone. So we have to force it by having the app do the functional equivalent with the readiness check.
I don't know of a tool that does a significantly better job at this without having other drawbacks and gotchas, and even if it did it doesn't void the value k8s brings.
I have my own set of gripes with software production engineering in general and specially with k8s, having seen first hand how much effort big corps have to put just to manage a cluster, but it's disrespectful to qualify this whole endeavour as disastrous.
People forget to enable pod readiness gates.
Unless something has changed since I last went digging into this. You will still have the ALB sending traffic to a pod that's in terminating state, unless you do the preStop bits I talked about in the top of the thread.
https://kubernetes-sigs.github.io/aws-load-balancer-controll...
Also allows graceful rollout of workload.
> You will still have the ALB sending traffic to a pod that's in terminating state
The controller watches endpoints and will remove your pod from target group on pod deletion.
You don't need the preStop scam as long as your workload respects SIGTERM and does lame-duck.
Calling it a scam is a bit much.
I think having to put the logic of how the load balancer works into the application is a crossing of concerns. This kind of orchestration does not belong in the app, it belongs in the supporting infrastructure.
The app should not need to know how the load balancer works with regards to scheduling.
The ALB Controller should be doing this. It does not, and so we use preStop until/unless the ALB controller figures it out.
Yes, the app needs to listen for SIGTERM and wait until it's outstanding requests are completed before exiting - but not more than that.
- so if pod goes to terminating state
- with gates enabled, alb controller should remove it from targets instantly coz it listens to k8s api pod changes stream ?
In my experience there was ALWAYS some delay even a small one in High Frequency systems which caused 500s.
Which we solved with internal apigateway, aws+iptables+cni was always causing issues in every setup without it.
Using SIGTERM is a problem because it conflicts with other behavior.
For instance, if you use SIGTERM for this then you have a potential for the app quitting during the preStop, which will be detected as a crash by Kube and so restart your app.
I don't think kubernetes restarts pods that have been marked for termination
We don't want to kill in-flight requests - terminating while a request is outstanding will result in clients connected to the ALB getting some HTTP 5xx response.
The AWS ALB Controller inside Kubernetes doesn't give us a nice way to specifically say "deregister this target"
The ALB will continue to send us traffic while we return 'healthy' to it's health checks.
So we need some way to signal the application to stop serving 'healthy' responses to the ALB Health Checks, which will force the ALB to mark us as unhealthy in the target group and stop sending us traffic.
SIGUSR1 was an otherwise unused signal that we can send to the application without impacting how other signals might be handled.
The ALB controller doesn't handle gracefully stopping traffic (by ensuring target group de-registration is complete) before allowing the pod to terminate.
Without a preStop, Kube immediately sends SIGTERM to your application.
sigterm_handler() {
make_healthcheck_fail();
sleep(20);
stop_web_server();
exit(0);
}
We do ALB Ingress to Istio Gateways
If you have a CDN, DDoS, or WAF you might be able to terminate there as well
Extra hop tho
Doesn't the kubernetes pod shutdown already mark the pod as not-ready before it calls the pre-stop hook?
One of my former co-workers went to a K8S shop, and longs for the simplicity of ECS.
No software is a panacea, but ECS seems to be one of those "it just works" technologies.
I was using K8s previously, and I’m currently using ECS in my current team, and I hate it. I would _much _ rather have K8s back. The UX is all over the place, none of my normal tooling works, deployment configs are so much worse than the K8s equivalent.
(And it’s free, if you don’t mind the mild lock-in).
So your application is now suddenly spread across multiple services, and you'll need an IaC tool like Terraform, etc.
The beauty (and the main reason we use K8s) is that everything is inside our cluster. We use cloudnative-pg, Redis pods, and RabbitMQ if needed, so everything is maintained in a GitOps project, and we have no IaC management overhead.
(We do manually provision S3 buckets for backups and object storage, though.)
However, GitOps is IaC, just by another name, so you actually do have IaC “overhead”.
Anything stateful is not allowed inside the cluster. PVs are annoying enough without having to manage a DB bolted onto what was originally designed for stateless web services.
Agreed running databases without operators that can handle replication, master promotion backups and PIT restore is super scary. Most of the modern operators support all of these operations.
But… if you are maintaining storage buckets and stuff elsewhere (to avoid accidental deletion etc, a worthy cause) then you are using terraform regardless. So adding RDS etc to the mix is not as tough as you make it sound.
I see both sides of the fence and both have their pros and cons.
If you have great operational experience with kube though I’d go all in on that. AWS bends you over with management fees… it’s far more affordable to run a DB, RMQ, etc on your own versus RDS, AMQ
[1]https://aws-controllers-k8s.github.io/community/docs/user-do...
The ugliness of k8s is that you're bringing your points of failure together into one, mega point of failure and complexity.
Final aside - you absolutely should be using IaC for any serious deployments. If you're using clickops or CLI then the context of the discussion is different and the same critera do not apply.
We have worked with Terraform modules before, but they quickly became difficult to manage.
Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection.
No they aren’t. All of the major IaC solutions (TF, CDK, etc) do ECS deployments directly through their own API, including with drift detection and updates.
Good for you for finding something that works, but it sounds like your advice related to IaC solutions is based on a misunderstanding of the benefits of IaC and the tools available.
I've never used Kubernetes myself, but ECS seems to "just work" for my use case of run a simple web app with autoscaling and no downtime.
The amount of companies who use K8s when they have no business nor technological justification for it is staggering. It is the number one blocker in moving to bare metal/on prem when costs become too much.
Yes, on prem has its gotchas just like the EKS deployment described in the post, but everything is so much simpler and straightforward it's much easier to grasp the on prem side of things.
Naively, I would think it be neutral, since I would assume that if a customer gets k8s running on-prem, then apps designed for running in k8s should have a straightforward migration path?
In cloud environments, it's pretty common that your cloud provider has specific implementations of Kubernetes objects, either by creating custom resources that you can make use of, or just building opinionated default instances of things like storage classes, load balancers, etc.
It's pretty easy to not think about the implementation details of, say, an object-storage-backed PVC until you need to do it in a K8s instance that doesn't already have your desired storage class. Then you've got to figure out how to map your simple-but-custom $thing from provider-managed to platform-managed. If you're moving into Rancher, for instance, it's relatively batteries-included, but there are definitely considerations you need to make for things like how machines are built from disk storage perspective and where longhorn drives are mapped, for instance.
It's like that for a ton of stuff, and a whole lot of the Kubernetes/OutsideInfra interface is like that. Networking, storage, maybe even certificate management, those all need considerations if you're migrating from cloud to on-prem.
I’ve ran 3 server k3s instances on bare metal and they work very well with little maintenance. I didn’t do anything special, and while it’s more complex than some ansible scripts and haproxy, I think the breadth of tooling makes it worth it.
Cost wise it's a no brainer. Three servers with 64 GB ECC and 6 cores for the price of three M5 larges. So 192 GB and 18 cores for the price of 24GB and 6 cores.
I think one of reason k8s can get a bad rap is how expensive it is to even approach doing it right with cloud hosting, but to me it seems like a perfect use case for bare metal where there is no built in orchestration.
DigitalOcean is just a hosting provider. You can buy just regular 'Droplets' (dedicated machine nodes) and host whatever you want on there, no serverless cloud junk. They also offer Managed Kubernetes. I use both.
Instead you can take k8s, deploy it to bare metal, and have a much much more power for a much lower cost. Of course this requires some technical knowledge, but the benefits are significant (lower costs, stable costs, no vendor lock-in, all the postgres extensions you want, response times halved, etc).
k8s smoothes over the vagaries of bare-metal very nicely.
If you'll excuse a quick plug for my work: We [1] offer a middle ground for this, whereby we do and manage all this for you. We take over all DevOps and infrastructure responsibility while also cutting spend by around 50%. (cloud hardware really is that expensive in comparison).
[1]: https://lithus.eu
>all the postgres extensions you want
You can run Postgres in any managed K8s environment (say AWS EKS) just fine and enable any extensions you want as well. Unless you're conflating managed Postgres solutions like RDS, which would imply that the only way to run databases is by using a managed service of your cloud of choice, which obviously isn't true.
You absolutely can do this, and we do ineed run Postgres in-cluster.
We generally see that people prefer a managed solution when it comes to operating their databases. Which means that when it comes to their (eg) AWS EKS clusters, they often use RDS rather than running the DB in-cluster.
Our service is also a managed service, and that comes with in-cluster databases. So clients still get a managed service, but without the limitations of (eg) RDS.
I remember a guy I used to work with telling me he'd been at a consulting shop and they used Kubernetes for everything - including static marketing sites. I assume it was a combination of resume and bill padding.
This guy told me it was just shameless over-engineering.
The AWS Load Balancer Controller uses readiness gates by default, exactly as described in the article. Am I missing something?
Edit: Ah, it's not by default, it requires a label in the namespace. I'd forgotten about this. To be fair though, the AWS docs tell you to add this label.
Damn it, now you've made me paranoid. I'll have to check the ELB logs for 502 errors during our deployment windows.
AWS uses them for all of their integrations and they're never not annoying.
At that point we already had a CRD used by most of out tenant apps, which deployed an opinionated (but generally flexible enough) full app stack (Deployment, Service, PodMonitor, many sane defaults for affinity/anti-affinity, etc, lots of which configurable, and other things).
Because we didn't have an opinion on what tenant apps would use in their containers, we needed a way to make the pre-stop sleep small but OS-agnostic.
We ended up with a 1 LOC (plus headers) C app that compiled to a tiny static binary. This was put in a ConfigMap, which the controller mounted on the Pod, from where it could be executed natively.
Perhaps not the most elegant solution, but a simple enough one that got the job done and was left alone with zero required maintenance for years - it might still be there to this day. It was quite fun to watch the reaction of new platform engineers the first time they'd come across it in the codebase. :D
I realized somewhat recently I could put my Nginx and PHP ini in a config map, that seems to work ok. Even that seems a bit dirty though, doesn't it base64 it and save it with all the other yaml configs? Doesn't seem like it's made for files
It does! It's mountable in the filesystem though. In this case, the data key is the filename, and its un-base64'd data, the file contents.
> Even that seems a bit dirty though
As I mentioned in the previous comment, "Perhaps not the most elegant solution" :D
It's been maintenance-free for years though, and since its introduction there were 0 rollout-related 502s.
20 years ago we used simple bash scripts using curl to do rest calls to take one host out of our load balancers, then scp to the host and shut down the app gracefully, and updated the app using scp again, then put it back into the load balancer after testing the host on its own. we had 4 or 5 scripts max, straightforward stuff..
They charge $$$ and you get downtime in this simple scenario ?
We had perfectly good rolling deploys before k8s came on the scene, but k8s insistence on a single-phase deployment process means we end up with this silly workaround.
I yelled into the void about this once and I was told that this was inevitable because it's an eventually consistent distributed system. I'm pretty sure it could still have had a 2 phase pod shutdown by encoding a timeout on the first stage. Sure, it would have made some internals require more complex state - but isn't that the point of k8s? Instead everyone has to rediscover the sleep hack over and over again.
> The AWS Load Balancer keeps sending new requests to the target for several seconds after the application is sent the termination signal!
And then concluded a wait is required…? Yes, traffic might not cease immediately, but you drain the connections to the load balancer, and then exit. A decent HTTP framework should be doing this by default on SIGTERM.
> I yelled into the void about this once and I was told that this was inevitable because it's an eventually consistent distributed system.
Yeah, I wouldn't agree with that either. A terminating pod is inherently "not ready", that not-ready state should cause the load balancer to remove it from rotation. Similarly, the pod itself can drain its connections to the load balancer. That could take time; there's always going to be some point at which you'd have to give up on a slowloris request.
How?
A load balancer can't send a new request on a connection that doesn't exist. (Existing connections being gracefully torn down as requests conclude on them & as the underlying protocol permits.) If it cannot open a connection to the backend (the backend should not allow new connections when the drain starts) then by definition new requests cannot end up at the backend.
This is how all pre-k8s rolling deployment systems I've used have worked.
So instead we move the logic to the application, and put a sleep in the shutdown phase to account for the time it takes for the load balancer to process/acknowledge the shutdown and stop routing new traffic to that node.
Again, I don't see why the sleep is required. You're removed from the load balancer when the last connection from the LB closes.
The pod delete event is sent out, and the load balancer and the pod itself both receive and react to it at the same time.
So unless the LB switchover is very quick, or the pod shutdown is slow - you get dropped requests - usually 502s.
Try googling for graceful k8s deploys and every article will say you have to put a preStop sleep in
Also, Kustomize can help with some of this since it will rotate the name of ConfigMaps so when any change happens, new ConfigMap, new Deployment.
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
You get observability pretty much for free, solution from 15 years ago were crap, remember Nagios and the like?
Old solutions would put trash all over the disk in /etc/. How many time did we have to ssh to fix / repair stuff?
All the health check / load balancer is also much better handled on Kubernetes.
Thus, this entire system of "Mark me not ready, wait for ALB/NLB to realize I'm not ready and stop sending traffic, wait for that to finish, terminate and Kubernetes continues with rollout."
You would have same problem if you just started up new images in autoscaling group and randomly SSH into old ones and running "shutdown -h now". ALB would be shocked by sudden departure of VMs and you would probably get traffic going to old VMs until health checks caught up.
EDIT: Azure/GCP have same issue if you use their provided ALBs.
Kubernetes is missing a mechanism for load balancing services (like ingress, gateways) to ack pods being marked as not ready before the pod itself is terminated.
https://argoproj.github.io/argo-rollouts/features/traffic-ma...
Is there a slick strategy for this? Is it possible to have minutes long pre-stop hooks? Is the only option to give client connections an abandon ship message and kick them out hopefully fast enough?
> Seamless Migrations with Zero Downtime
(I don't work for them but they are friends ;))