Common mistakes using Kubernetes
blog.pipetail.io
blog.pipetail.io
When I was writing my book my editor asked me to remove any writing about mistakes and changes I made in the project for each chapter. I had a bug that appeared and I wanted to write about how I determined that and fixed it. They said the reader wants to see an expert talking, as if experts never make mistakes or need to shift from one tact to another.
But, I find I learn the most from explanations that share how your mental model was wrong initially and how you figured it out and how you did it "more right" the next time.
That's really how people build things.
Your editor was very fucking wrong.
I think that'd be more of the author's choice, instead of the editor.
I've never actually heard that one before, but it's so very, very true.
The editor is completely right in what they were saying. You just want them to be wrong, because you'd prefer to live in the fantasy world where they are wrong.
Let's say you go to get a surgery. You don't want the doctor to tell you about all the times they fucked up and what the awful consequences were. It doesn't matter that they're probably a better surgeon now, having learned from their mistakes. Psychologically, you need that person with the sharp tool poking around inside your body to be a superhuman.
To a lesser degree, the same is true for any expert. Of course everybody makes mistakes. Notice the de-personalization in the word "everybody". You can talk about the mistakes everybody makes, or those ones that many people make. If you talk about your own mistakes however, you lose the superhuman status. There may be a few situations where that somehow helps you, but not when you want to sell books.
That they won’t tell me (the patient) is quite a different question from whether they got the material from someone more experienced in their primary or continuing medical education.
Shouldn't that be more like "Let's say you're learning to be a surgeon."?
For that situation, the person they're learning from discussing problems they hit and how they solved them does sound like it would be very useful.
No. It is an exaggerated example, because I am illustrating a psychological effect.
The point is that you are consulting an expert, and you don't want that expert to tell you about all their mistakes, because your mind is now occupied with doubting that expert. It's not rational.
> For that situation, the person they're learning from discussing problems they hit and how they solved them does sound like it would be very useful.
It's useful to present common mistakes, but mentioning who made the mistake is both irrelevant and damaging.
Sometimes books are written for entry level people (to the technology or topic) and they're written in an authoritative tone. But I work with people who write books. And I read their books. I don't read them like some kind of expert oracle distributing the blessed texts. I read them as a work of documentation from a friend or colleague, so I don't really feel this veneer of expertise is necessary.
Sometimes the experts try hard to scrape away the veneer of expertise by adding side fluff like in Learn you a Haskell for Great Good. Or, Alex Crichton, who is undoubtedly an expert in Rust and compiler technology giving this talk where he keeps saying things like "it does a process called quasiquoting which doesn't make any sense to me", "I don't know what hygiene means"
Yes, you feel that way. You feel like you're a rational person who values humility, whose judgement isn't clouded by the effects that I am describing.
Well, maybe you really are that person. It doesn't matter. We cannot assume that the buyers are such noble beings.
In the situation where I'm consulting an expert - hired to fix a problem or find a solution - then I'd tend to agree with you.
But when I'm reading text from a person in my field in a subject area of interest (eg learning from a book), then I definitely want to hear of things that went wrong, and how they fixed them.
I think they're two different situations, and the one you described above isn't the correct one for a person who's bought a book to learn from.
It could just be us valuing different things in our learning though. :)
I like this analogy and I agree that the previous person saying the publisher was right, completely missed the point.
I also think this is the way the majority of tech books are written.
Can you think of another where the author goes from mistake to mistake and then finally gets it right?
I believe there is a space in tech writing for this kind of writing, but it is not something traditional book publishers believe.
This was an O'Reilly book by the way, with really good editors and a really good editoral process.
That editor was right most of the time, IMHO.
Not a book, but Raymond Hettinger often presents in this way and it's fantastic: https://www.youtube.com/watch?v=wf-BqAjZb8M
Little Lisper, Little Scheme is a conversation style that discusses a lot of false starts.
{Coders|Founders} at Work has a lot of frank talk about early mistakes people make and their pivots.
Tact means skill in dealing with other people, particularly in sensitive situations.
Tack has to do with sailing boats into the wind, and crucially "changing tack" means changing direction.
I was told not to mention the caveats, instead, render a confident image for the team many times in my career.
It's like doctors suggest their patients to use some drugs without mentioning any side effects.
It reminds me of the video[0] which asks the developer to draw red lines with blue ink, while the project manager keeps pushing the developer like "You're the expert, of course you can do draw red lines with blue ink!".
One language that didn't go how I expected it was applescript.
The book on applescript (from o'reilly I believe) was necessarily different.
Instead writing "normal" top-down programs, applescript hooks into the OS from the side.
Many books can concentrate on what you can do, but the applescript had to do everything by example. This is because all of apple's applications expose interfaces that you have to figure out on the fly.
I had to learn a different way from that book, because it necessarily concentrated on "how" instead of "what".
Your editor is right. Not because this information is not interesting.
But it is distracting and off-topic to go off a tangent every now and then. Especially for someone just learning the stuff, it will confuse them more than anything else.
Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.
- Azure (still?) offers free clusters.
- GCP offers one free single-zone cluster.
If you need more than DO can offer in terms of compute instances, you can probably afford GKE/EKS, which is around 75$/month.
---
Running highly-available control plane (K8s masters & etcd) by yourself is NOT cheaper than using EKS.
To achive high availability, EKS runs 3 masters and 3 etcd instances, in different availability zones. Provisioning 3 t3.medium instances (4 GB of memory and 2 CPUs) would cost the same as a completely managed EKS.
Not to mention the manual work you need to setup, maintain and upgrade such instances.
If someone knows about a reliable managed kubernetes, please let me know.
I might be remembering wrongly, but I think Azure actually used to offer multiple managed orchestration platforms, including DCOS and Swarm, before they went all-in on k8s.
If yes, Kubernetes might make sense for you, if your company is large enough that a PaaS like Heroku is prohibitively expensive.
If no, you don't want Kubernetes.
https://twitter.com/kelseyhightower/status/93525292372179353...
Amazon's Elastic Container Service (ECS) on Fargate deployment type is probably a better option much of the time. Until you maintain your own k8s cluster (including the hosted variants on AWS, GCP, etc.) you might not realize how complex configuring k8s is.
While the AWS's ECS service may be more limited, I've found it leaves less room to do the wrong thing. Unfortunately, the documentation on ECS and the community support is inferior to k8s, but I'll accept that if I don't have to spend a whole day researching what service mesh to use with k8s and how to configure a load balancer and SSL certs.
But I generally agree with the premise that ECS is a better starting point for people just dipping their toe into containers.
ECS (with Fargate or EC2) is awesome but one might argue that it is still an overkill for a lot of people.
Some people look for scaling, immutability, effective resource utilization and they think they NEED containers. That is not entirely true. A lot of people would do totally ok with one EC2 instance.
Either way, kubernetes is awesome, there is a big community behind it maintaining a lot of tooling, docs, how-to guides, etc. That is very valuable for a lot of people too :)
Sorry—my comment was written poorly. I think the article is good as is and agree k8s or not is a different topic. My comment intended to be a reminder for all of us reading it that we should ask more questions before choosing a technology.
I’ve found a trend in engineering (including with myself), we sometimes forget to take a step back and see the forest.
x1e.16xlarge, 64 CPU, 2TB of RAM, $10k per month.
1. There. Is. No. Machine. (Insert matrix meme here.) Before you open up your cluster to the rest of company, drill it down to them. Maybe even create a Google Form where they have to sign "I hereby acknowledge that there is no machine in k8s and any attempt to tie my job to a particular machine means a broken config by definition."
2. Thanks to 1, don't let anyone use hostNetwork, hostIPC, hostPID, hostPorts, host whatever, unless you have a really good reason to (with explicit approval process).
3. Don't let anybody start a job without memory/CPU limit. Make sure they understand that, if the job goes over the memory limit, it dies, and it's not k8s admin's problem.
4. You can't log anything into the pod - when the pod dies the log is gone. You can't log into the machine, either (see 1). Therefore, you really need some kind of logging framework that takes the log from your pod and saves it, in its raw form, somewhere safe (like S3). I don't know if there's any such framework, but there had better be.
5. Make sure every manual operation is logged (who did what to which job when), unless you like asking "@here Does anybody know who owns fooservice?" every month.
6. Kubernetes is not magic: if it takes thirty minutes to provision your service, fix that, instead of moving thirty minutes of manual provisioning into k8s and somehow expect it to be magically reliable.
7. Don't bring in existing dependencies uncritically. If your job connects to a zookeeper server to find out its peers, don't bring it into k8s, but rewrite it to use k8s service instead.
8. Take extra extra care when writing down your first job specification, because there are a lot of yaml files to write, and people will just copy what's already there. If your first k8s job mounts host /tmp directory just because you were testing something and forgot to delete the line, soon you will have fifty jobs all mounting host /tmp directory. Good luck figuring out which job actually needs it then.
Yeah, again, I'm by no means an expert - I'm not even an admin, so just consider the list as a rambling of some poor soul who has seen some stuff. Here be dragons, have fun.
This piece of advice is so underrated, it's hard to put into words. Every senior developer/architect should know that.
7. Sadly if existing software can't run in kubernetes, this severely limit the benefits of having kubernetes, why use something that can't be used? If the jobs already have service discovery, they might be better off running on hostNetwork or whatever allow them to work as is.
From what I've seen, Kubernetes is a quintessential Google product - it has a very particular idea of how jobs should be run, and the farther you stray away from it, the more it will cost your sanity. If that curtails the benefits of k8s for you, then yes, I think you should revisit whether you really need k8s, based on that limitation.
Just my two cents.
Actually, zookeeper has API to register only when ready and to listen to events/changes. Not sure kubernetes has equivalent stable API so might be hard to port over.
A rewrite might be a solution, except it's not because it's doomed to fail. We're discussing service discovery, which implies multiple clients and servers, probably managed by different teams and written in different languages. The odds of completing a coordinated rewrite effort are abysmal. ^^
Well. I am thinking out loud. It's a real problem my company was facing. We've got kubernetes clusters that are supposed to run applications and we've got apps using zookeeper that can't run in kubernetes because it breaks the service discovery. I will eventually have to hint people how to make it happen, after one year of kubernetes hardly going anywhere.
We solve #2 (and more) by having a highly restrictive default PodSecurityPolicy. We started with a combination of GCE and OpenShift default examples (although we use neither) which are published on github. PSP lets admins and security relax a little at night.
Anything #3 is solved by setting highly restrictive LimitRanges in each namespace that must be overridden in the deployment specs.
It's not mandatory that you override the defaults, but if you don't you're going to get poor performance. If you're fine with that the admins are also fine with that.
Sometimes some occasional throttling during spikes is totally acceptable - it all just depends on whether the app actually needs maximum performance. There are many other apps in the cluster that DO, and in the grand scheme of things they benefit by having the ones that DON'T get throttled.
Apparently Borg handles CPU differently (and better) than k8s in the multitenancy model, but this is the best "poor man's" borg I can come up with and it works for us.
I've created a static code analyzer for Kubernetes objects, called kube-score, that can identify and prevent many of these issues. It checks for resource limits, probes, podAntiAffinities and much more.
kustomize build | kube-score score -It's important to understand that CPU resource requests are used for scheduling and not for limiting. As the author suggests, this can be an issue when there is CPU contention, but on the other hand, it might not be. That's because memory limits are even more important than CPU requests when scheduling: most applications use far more memory as a proportion of overall host resources than CPU.
Let's take an example. Suppose we have a 64GB worker node with 8 CPUs in it. Now suppose we have a number of pods to schedule on it, each with a memory limit of 2GB and a CPU request of 1 millicore (0.001CPU). On this node, we will be able to accommodate 32 such pods.
Now suppose one of the pods gets busy. This pod can have all the idle CPU it wants! That's because it's a request and not a limit.
Now suppose all of the pods become fully CPU contended. The way the Linux scheduler works is that it will use the CPU request as a relative weight with respect to the other processes in the parent cgroup. It doesn't matter that they're small as an absolute value; what matters is their relative proportion. So if they're all 1 millicore, they will all get equal time. In this example, we have 32 pods and 8 CPUs, so under full contention, each will get 0.25 CPU shares.
So when I talk to customers about resource planning, I actually usually recommend that they start with low CPU reservation, and optimize for memory consumption until their workloads dictate otherwise. It does happen that particularly greedy pods are out there, but that's not the typical case - and for those that are, they will often allocate all of a worker's CPUs in which case you might as well dedicate nodes to them and forget about how to micromanage the situation.
A lot of this depends on policy and cooperation, which is true for any multitenant system. If the policy is that nobody requests CPU, then the behavior will be like an ordinary shared Linux server under load - the scheduler will manage it as fairly as possible. OTOH, if there are pods that are greedy and pods that are parsimonious in terms of their requests, the greedy pods will get the lion's share of the resources if it needs them.
The flip side of overallocating CPU requests is cost. This value is subtracted from the available resources, making the node unavailable to do other useful work. Most of the time I see customers making the opposite mistake - overallocating CPU requests so much that their overall CPU utilization is well under 25% during peak periods.
The example you are describing is probably not super common but I will try to rephrase my blogpost so that it reflects this comment:)
K8s is a powerful and complex tool that should only be used when needed. IMO you should be wary of using it if you're trying to host less than a dozen applications - unless it's for learning/testing purposes.
It's a complex beast with many footguns for the uninitiated. For those with the right problems, motivations and skillsets, k8s is the holy grail of scale and automation.
I would also not recommend it for hosting a WordPress site, or a simple CRUD app.
However, when you get to the level where autoscaling is required, and where you are deploying multiple services, managed Kubernetes is not such a bad idea.
Using EKS (especially with Fargate) on AWS is not much harder than figuring out and properly utilizing EC2/ASG/ELB. GKE or DigitalOcean offerings seem to be even easier to use and understand.
You could look at the big cloud providers directly. OpenShift [I work at Red Hat] also has a few different types of managed offerings.
Also, to take full benefit of the QoS you need to configure the Kubelet with "--cpu-manager-policy static"[0].
[0]: https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-...
...
requiredDuringSchedulingIgnoredDuringExecution:
...
Kubernetes sometimes shows its Java roots.
printf isn't named printIntoBufferWhichMayNotFlushUntilLinefeed, and people are fine with it.
Citation needed.
AFAIR Borg is implemented in C++ and k8s has been implemented in Go from day 0.
Am I missing some crucial steps in k8s's history?
https://kubernetes.io/blog/2018/06/06/4-years-of-k8s/
> Concretely, Kubernetes started as some prototypes from Brendan Burns combined with ongoing work from me and Craig McLuckie to better align the internal Google experience with the Google Cloud experience. Brendan, Craig, and I really wanted people to use this, so we made the case to build out this prototype as an open source project that would bring the best ideas from Borg out into the open.
> After we got the nod, it was time to actually build the system. We took Brendan’s prototype (in Java), rewrote it in Go, and built just enough to get the core ideas across
The submitted title was "10 most common mistakes using kubernetes", HN's software correctly chopped off the "10", but then you added it back. Submitters are welcome to edit titles when the software gets things wrong, but not to reverse things it got right.
This is what I'm trying to convince my current company about. They want everything in a single cluster (prod, test, stage, qa).
Of course self hosting makes this more difficult to justify, since it is additional expenses for more machines.
Seems like a perfect use-case for Cluster API: https://cluster-api.sigs.k8s.io/user/quick-start.html
Have one global "mgmt cluster" with several workload clusters
Ah interesting! We'll have to look into that.
When your doing rolling upgrades it can signal your app is ready to take traffic.
Those are the main uses.
For liveness probe I guess you could check if your service is accepting TCP connections? I don’t think there should ever be a reason for your service to outright refuse connections unless the main service process has crashed (in which case it’s best to let Kubernetes restart the container instead of having a recovery mechanism inside the container itself like supervisord or daemon tools).
If the underlying dependencies aren't working, can a pod actually be considered ready and able to serve traffic? For example, if database calls are essential to a pod being functional and the pod can't communicate with the database, should the pod actually be eligible for traffic?
> Do not fail either of the probes if any of your shared dependencies is down, it would cause cascading failure of all the pods.
The idea would be that the downstream dependencies have their own probes and if they fail they will get restarted in isolation without touching the services that depend on them (that are only temporarily degraded because of the dependency failure and will recover as soon as the dependency is fixed).
From the article: > The other one is to tell if during a pod's life the pod becomes too hot handling too much traffic (or an expensive computation) so that we don't send her more work to do and let her cool down, then the readiness probe succeeds and we start sending in more traffic again.
Well... maybe. Is it a routine occurrence that an individual Pod becomes "too hot"? If your load balancer can retry a request on, say, a 503 Service Unavailable, you may be better off relying on that retry combined with CPU-based autoscaling to add another Pod (it's simpler, tradeoff is the load balancer may spend too much time retrying).
If you can't or don't want to add additional Pods, then your client is going to see that 503 (or similar) anyway. I'd say, then, that the point of a Pod claiming it's "not ready" to get itself removed from the load balanced pool is to allow the load balancer to more quickly find an available Pod, but this adds complexity and may be irrelevant if you run enough Pods to have some overhead capacity.
A Rails app is a bit different from a node/go/java app in that (typically at least, if you're using Unicorn or other forking servers) each individual Pod can only handle a limited number of concurrent requests (8, 16, whatever it is). It's more likely then that any given Pod is at capacity.
But, liveness/readiness are not so simple. If these probes go through the main application stack, then they're tying up one of the precious few worker processes, even if only momentarily. I haven't worked with Ruby in a number of years, but I remember running a webrick server in the unicorn master process, separate from the main app stack, to respond to these checks. But I did not implement a readiness check that tracked the number of requests and reported "not ready" if all the workers are busy.
I have used Kubernetes pretty heavily in the past, and didn't know about PodDisruptionBudget.
Why isn't this the default behavior? Why don't I have to go in and tell it that it's okay to have multiple instances on the same node? Why? So that I somehow feel like I've contributed to the whole process by fixing something that never should break in the first place?
I know of a few pieces of code where I definitely want to run N copies on one machine, but for all of the rest? Why am I even running 2 copies if they're just going to compete for resources?
When you are talking about the realm of the possible, you provide settings that allow you to reach the scenarios that you feel are reasonable, desirable, or lucrative (or commonly enough, some happy combination of the three).
Defaults are the realm of the probable. And nobody is requisitioning a 192 core machine without a good bit of due diligence, which would include deciding how to set server affinity.
Do you think that batch processing is the default activity in Kubernetes, or something that people find after they are familiar with the system?
Why does the resource weight have anything to do with the choice of defaults? Settings don't care how often they are read, they only care how often they are set. Large jobs use a disproportionate amount of total resources, sure, but they are tiny uptick in total configuration.
The stakes are higher, but so is the 'budget' for getting things right. I can deploy 5 servers and just wait to see what happens. If I'm doing an overnight job to process a billion records, I'd better be doing some due diligence beforehand, or I have nobody to blame but me. And the failure mode here is that I didn't spend money fast enough to get the job done.
With the current defaults what happens is I blow my monthly budget in one night. Which is very convenient for the vendor, but not convenient for my company.
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair
If configured as suggested, and for some reason you lose enoguh nodes in your cluster for not having single node for each of your replicas, you will have less replicas than you intended, since the scheduler can't schedule a pod to a node where identical pod is already running.
Additionally, the affinity and anti affinity features are costly from the cluster perspective, so the configuration recommended by the author cost you performance.
And why isn't Kubernetes doing the obvious thing and spread apart your pods? well, it's simple - it does the right thing:
https://v1-15.docs.kubernetes.io/docs/concepts/scheduling/ku...
The first scoring parameter is SelectorSpreadPriority.
Maybe on GCP (I don't see a lot of companies on GCP) it makes sense, but ECS is AWS native and on bare hardware I immediately go to docker swarm since it ships with the container runtime (instead of a bolted on sidecar container thing).
I like the primitives and features of Kubernetes, but the implementation doesn't give me warm fuzzies and it always gets passed over for safer bets for me. Even very early on in it's development I always went to Mesos over Kubernetes (though Mesos is P dead at this point).
Common mistakes: using Kubernetes
For example on the liveness / readiness probe item, the article says,
> “ The other one is to tell if during a pod's life the pod becomes too hot handling too much traffic (or an expensive computation) so that we don't send her more work to do and let her cool down, then the readiness probe succeeds and we start sending in more traffic again.”
But this is often a very bad idea and masks long term errors in underprovisioning a service.
If the contention of readiness / liveness checks vs real traffic is ever resulting in congestion, you need the failure of the checks to surface it so you can increase resources. If you set things up so this failure won’t surface, like allowing the readiness check to take that pod out of service until the congestion subsides, you’re only hurting yourself by masking the issue. It basically means your readiness check is like a latency exception handler outside the application, very bad idea.
The other item that is way more complicated than it seems is the issue about IAM roles / service accounts instead of single shared credentials.
In cases where your company has an enterprise security team that creates extremely low-friction tools to generate service account credentials and inject them, then sure, I would agree it’s a best practice to ruthlessly split the credentialing of every application to a shared resource, so you can isolate access and revoking.
But if you are on some application team and your company doesn’t have a mature enough security tooling setup managed by a separate security team, this can become a bad idea.
It can lead to superlinear growth in secrets management as there will be manual service account creation and credential propagation overhead for every separate application. Non-security engineers will store things in a password manager, copy/paste into some CI/CD tool, embed credentials as ENV permanently in a container, etc., all because they can’t create and maintain the end to end service account credential tools in addition to their job as an application team engineer. It’s something they think about twice per year and need off their plate immediately to move on to other work.
Across teams it means you end up with 20 different team-specific ways to cope with rapid growth of service accounts, leading to an even worse security surface area, risk of credential-based outages, omission of important testing because ensuring ability to impersonate the right service account at the right place is too hard, etc.
Very often it is a real trade-off to consider that one single service account credential that has just one way to be injected for every service is safer in the bigger picture.
Yes it means a credential issue for any service becomes an issue for all, and this is a risk and you want automated tooling to mitigate it, but it very often will be less of a risk than insisting on a parochial best practice of individual service account credentials, resulting in much worse and less auditable secrets workflows overall unless it is completely owned and operated by a central security team in such a way that it doesn’t create any approval delays or workflow friction for application teams.
This is an argument for making your liveness probe == readiness probe. It should just check pod availability in a minimal way, and if continuing to send the pod traffic based on this indicator turns out bad because of congestion, you want to see that causing errors and react, not let the scheduler take it out of service for new traffic.
You want liveness & readiness to check the same thing, and it should be a non-trivial check of service health that is also very low latency. And as long as that check is passing, keep sending traffic.
When the check fails, it should always be for a “hard down” reason that tells you the pod could not, regardless of traffic levels, accept traffic because it’s fundamentally internally down.
That’s just bad engineering. Really, one should want the pager to go off for that and be really pedantic to actually sniff out the root cause and actually fix it.
Hiding that type of issue by letting something like liveness/readiness policy tacitly conceal it is just going to result in a far worse or more systemic issue later with far worse pager disruptions to your life.
You’re skipping flossing every now and then only to need serious root canals later.
I feel like it's a weekly occurrence now where I hear of a startup launching their mvp on kubernetes having spent 8 months too long on Dev as a result.
The other day in an interview someone bragged to me how he had convinced his team to spend 12 months moving to K8s. Upper management thought it was a waste of time but eventually agreed. I asked him if there were any measurable benefits and he said no.
I totally understands why Google needs it. Do you?
These days, some people are like "Dude, if you don't have one cluster per AWS availability zone per each environment, you're doing it wrong." Why, just why.
This is absolutely becoming a tiresome trope. K8s is a huge benefit to tons of companies and none of them are Google.
Yes some people are using K8s when they don't need it. Just like many are using cloud managed services when they don't need them. Or vms. Or insert any technology here.
This article has nothing to do with whether K8s fits some particular use case but may be of help (although I disagree with the entire section on resources which reflects a lack of long term experience with K8s in production) to those who do want to use it.
You're the 10th person in this thread saying the same thing and it doesn't appear you even have that much experience with operations in general.
Sorry to go off on you but I'm really seeing these types of tropes and quick depthless one liner comments and offtopic snipes lately as the downfall of the hn comment section.