The hater's guide to Kubernetes
paulbutler.org
paulbutler.org
I remember back when the Cloud first started getting a foothold that what people was drawn to was that it would enable reducing complexity of managing the most frustrating things like the load-balancer and the database, albeit at a price of course, but it was still worth it.
Stateless app servers however, was certainly not a large maintenance problem. But somehow we've managed to squeeze in things like k8s in the there anyway, we just needed to evangelize microservices to create a problem that didn't exist before. Now that this is part of the "culture" it's hard to even get beyond hand-wavy rationalizations that microservices is a must, assumingly because it's the initial spark that triggered the whole chain reaction of complexity.
I'm constantly fascinated how people handwaivingly underestimate the cost and headaches of actually running on prem global infrastructure.
To call a halt to your constant fascination: they don't all have that problem. They still get the complexity of cloudy things regardless when they use one.
Cloudflare zero trust for free is a huge timesaver
Two datacenters on opposite sides of the US from different providers will get you more uptime than a cloud provider and is super simple.
It's totally worth it for some companies to do that, but you need to have some serious size to be concerned with spending your efforts on lowering your AWS bill by introducing details like that into your own organization when you could alternatively spend those dollars to make your core business run better. Usually your efforts are better spent on the latter unless you are Netflix or Amazon or Google.
There are multiple providers that offer VPS and ingress/egress for a fraction of the cost of public clouds and they mostly have good uptime.
The uptime in these DCs is very good (certainly better than AWS's us-east-1), and you get a very good price with tons of bandwidth. Most datacenter and colo providers can do this now.
I think people believe that "on prem" means actually racking the servers in your closet, but you can get datacenter space with fantastic power, cooling, and security almost anywhere these days.
That's because that is what on prem means. What you're describing is colocating.
On top is AWS lambda or something where you are completely removed from the actual hardware that's running your code.
At the bottom is a free acre of land where you start construction and talk to utilities to get electricity and water there. You build your own data center, hire people to run and extend it, etc.
There is tons of space in between where compromises are made by either paying a provider to do something for you or doing it yourself. Is somebody from the datacenter where you rented a rack or two going in and pressing a reset button after you called them a form of cloud automation? How about you renting a root VM at Hetzner? Is that VM on prem? People who paint these tradeoffs in a black and white matter and don't acknowledge that there are different choices for different companies and scenarios are not doing the discussion a service.
On the other hand, somebody who built their business on AppEngine or CloudFlare workes could look at that other company who is renting a pet pool of EC2 instances and ask if they are even in the cloud or if they are just simulating on-prem.
Which, first order approximated, is a function of (1) how big a company you are (aka "Can you even afford to hire two people to just do X?") and (2) how competitive the market is for X.
Colo and dedicated VMs are so reasonably priced because it's a standardized, highly-competive market.
Similarly, certain managed cloud services are ridiculously expensive because they have a locked-in customer base.
Which would suggest outsourcing components that have maximum vendor competition and standardization, as they're going to be offered at the lowest margin.
What kind of weird stuff are we talking?
Essentially very high security and throughput TRNG servers (with cryptographic signing and the like).
It reads like propaganda sponsored by the clouds. Scaremongering.
Clouds are incredibly lucrative.
But don't worry. You can make the prices more reasonable by making a 3-year commitment to run old outdated hardware.
Vertical integration is a widely known and understood business strategy - running your own infrastructure will help you reclaim the cloud margins back for yourself.
You can do it as a one man band or a huge multinational.
I use hotels and other rental offerings, including the cloud. But I when it is advantageous to do so, I buy and own. Even though it comes with maintenance burdens.
“What alternative is there to living in a hotel if you aren’t a carpenter!?!?!”
Of course there are plenty of scenarios where latency does not matter at all.
And many people who aren’t impulse buying will not stick around on slow sites.
Evidence: my wife buying our groceries for delivery at home. We have 4-5 choices in our city. All their websites are slow as hell, and I mean adding an item to a cart takes good 5-10 seconds. Search takes 20+ seconds.
She curses at them every time yet there's nothing we can do. The alternative is both of us to travel by foot 20 minutes to the local mall and wait on queues. 2-3 times a week. She figured the slow websites are the lesser evil.
When I go to book my colonoscopy on my hospital’s reservation system, I don’t bail out and look for a new doctor if it takes me 10 tries.
There are very few businesses where UX latency at the sub second level matters and the ones that do are not the ones you want to be in.
I also know that user experience of any application suffers a lot when latencies are high. Your point seems to be that there is lots of software that doesn't care about its user experience (mostly because the people making buying decisions are not the people suffering from those decisions) and that's a fair point, but I don't think that is a great business strategy for any software business.
Of course there are lots of scenarios where latency literally doesn't matter at all.
Any evidence to back this up. Because on the surface seems like a ridiculous statement.
When they pay their local utility bills, is that international? How about paying their rent? How about filing their state taxes? How about ordering from local suppliers?
Very little of the world is international business. That tiny slice that is just dominates the zeitgeist because it’s international.
For some examples of things well known that absolutely don’t need global data centers:
- airbus and Boeing
- Coca-Cola
- Marriott and Hilton
- the entire US federal government (apart from some maybe military applications)
- McDonald’s
The list goes on forever because it’s literally nearly every business. Unless you’re in real time markets or operating store fronts globally where latency hurts sales, putting up regions all over the world is a complete and utter waste of money.
Making global regions as easy as a click of a button was one of the greatest marketing ploys of cloud providers to date.
“Of course Good Will needs a Singapore data center!? How will we meet our P99 goals otherwise?”
My siblings work there and their work information system is not so bad but definitely it'd be totally frustrating if it didn't run on AWS in EU.
There is absolutely zero reason for a kiosk to touch 'a global data center'. No, not even for a payment, because it's just asks the payment terminal if the payment succeeded or not.
So McDonald's should be dispatching kiosk admins all around the world? That's very much not eco friendly. And of course, expensive... And a total nightmare to manage. K8s and AWS is a night walk through the rose garden compared to that.
> No, not even for a payment, because it's just asks the payment terminal if the payment succeeded or not.
Yeah sure, works great as long as the kiosk doesn't crash during the payment.
And they don't. Why the hell do you pull this nonsense?
In case you don't know (looks like you don't) these are Windows machines[0]. Any Windows machine is capable running IIS or even Apache on it. No need for the servers for managers to manage. Just effing serve it locally, if you can't provide each McD with a mini box what is managed by the central IT.
> should be dispatching kiosk admins all around the world
LOL
> K8s and AWS is a night walk through the rose garden compared to that.
You need a sysadmin to teach you how to build robust and resilient local applications and services without being Eco-unfriendly and without racking billions in AWS bills.
> Yeah sure, works great as long as the kiosk doesn't crash during the payment.
Do you understand what it is the payment terminal which is processing the payment and you can't offload reading CC/NFC to the 'cloud'?
And finally, there is already 'IT' infrastructure in each McD: Ethernet switches, UPSes, wireless AP and maybe a controller, menu and order screens at the counter, networked PoS and whatever else. If you claim what restaurant managers manage all these then you don't really know anything about nor retail nor IT. Or just bickering in a bad faith.
[0] and if in your case they are Linux ones... do I even need to continue?
I built similar kind of self service box (self checkout) for a local store chain. I know the problem well. Main problems are costs, system administration and management. Our solution is just a simple Chrome kiosk mode browser window too. It's a smart solution.
We run it on AWS because no reason not to - simply pushing the SPA to S3 behind Cloudfront beats any other kind of deployment/hosting method. Some backend stuff runs in Lambda, some runs in ECS containers. All data in a managed RDS DB. Easy to use, easy to maintain, easy to upgrade, easy to deploy, easy to scale from 0 to several thousands kiosks...
I used to be a Windows admin (though admittedly the last Windows I managed was 2003 R2). Just the word IIS makes my neck hair stand up.
Especially if it should be holding payment data... Huh, damn. Wow.
You'd need sync to a global service anyways - it's a multi store chain. Your suggestion just makes everything harder and more convoluted - and it's just wrong, the payment terminal has its own network connection and doesn't need the kiosk to work at all. The kiosk just initiates the payment but the rest of it is handled out of it - and the kiosk waits for central service to confirm the payment. It's bullshit to connect the kiosk to the bank, we support many banks and keep adding support for more, and what if it's stolen, should we always be ready to rotate the certs on 1000s devices - the bank doesn't have appropriate api, so it's either that or again, a central service? Nah, we just turn it off in our admin panel.
I also used to be a Linux sys admin way before clouds. I built one of the first cloud services in my country to solve the problems of that. The company ultimately was out competed - but not by traditional hosting. By the big public clouds. Apparently the problems are real.
BTW coincidentally I'm just about to launch a new cloud platform. Nothing fancy really, but it's built on dedicated servers for performance. Last 3 weeks I spent working on stuff I could've done with 100 lines of Terraform/Pulumi with AWS. Maybe a proper sys admin like you could teach me? I am not happy with it at all, it's a major headache and I'm considering a hybrid setup because I just don't want to lose sleep over customer data and site availability.
The only motivation would be latency, but you could have specialized services that run at the edge, if that's so important, for example if payment verification should take 500ms instead of 3000ms.
But you could also just rewrite the protocol to have less back-and-forth sequential data exchange, which is a smarter approach.
But what other solution allows you to:
* declarative define your infrastructure
* gives you load balancing, automatic recovery and scaling
* provides great observability into your whole stack (kubectl, k9s, ...)
* has a huge amount of pre-packaged software available (helm charts)
* and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
The answer is: there isn't any.
Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.
But it's still by far the best we've got.
You’re right that kubernetes is a bit batteries included, and for that its tempting to take it off the shelf because it “does a lot of needed things”, but you don’t need one tool to do all of those things.
It is ok to have domain specific processes or utilities to solve those.
Kubernetes is currently the best way to wrap up workloads in a cloud agnostic way. I've written dozens of services for K8s using different deployment mechanisms (Helm, Carvel's kapp, Flux, Kustomize) and I can run them just as easily in my home K8s cluster and in GCP. It's honestly incredible; I don't know of any other cloud tech that lets me do that.
One thing I think a lot of people miss too, is how good the concepts around Operators in Kubernetes are. It's hard to see unless you've written some yourself, but the theory around how operators work is very reminiscent of reactive coding in front end frameworks (or robotics closed loop control, what they were originally inspired by). When written well they're extremely resilient and incredibly powerful, and a lot of that power comes from etcd and the established patterns they're written with.
I think Kubernetes is really painful sometimes, and huge parts of it aren't great due to limitations of the language it's written in; but I also think it's the best thing available that I can run locally and in a cloud with a FOSS license.
I'll give you the benefit of the doubt here and say that some of the basics are indeed cloud agnostic.
However, it's plain for many or most to see that outside of extremely "toy" workloads you will be learning a specific "flavour" of Kubernetes. EKS/GKE/AKS etc; They have, at minimum, custom resource definitions to handle a lot of things and at their worst have implementation specific (hidden) details between equivalent things (persistent volume claims on AWS vs GCP for example is quite substantially different).
Thinking about it; the things I see as very cloud agnostic: Horizontal pod autoscaling, Node autoscaling, Layer 4 loadbalancing, Persistent volumes, Volume snapshots, Certificate managment, External DNS, External secrets, Ingress (when run in cluster, not through a cloud service),
That ends up covering a huge swath of my usecases, probably 80-90%. The main pain points I usually run into are: IAM, Trying to use cloud layer 7 ingress (app loadbalancers?)
I totally agree the underlying implementation if resources can be very different, but that's not the fault of Kubernetes; it's an issue with the implementation from the operator of the K8s cluster. All abstractions are going to be leaky at this level. But for PVCs I feel like storageclasses capture that well, and can be used to pick the level of performance you need per cloud; without having to rewrite the common provision of block device.
When I say something is easy to move, I mean that when I build on top of it, it's easy for users to run it in their cloud of choice with changes in config. It also means I have flexibility with where I choose to run something after I've developed it. For example I develop most stuff against minikube, then deploy it to GCP or a local production k8s. If I was using Terraform I couldn't do that.
If you'll excuse a slight digression, but I think there's a tendency atm to rather pay $1 in extra complexity a 100 times over time, than pay a $5 one-time fee. Like if repeating something similar twice - even if it's easy and not really a lot of effort - is a sign of failure and thus unbearable.
* your stack almost always ends up closely tied to one cloud provider. I've done and seen cloud migrations. They are so painful and costly that they often just aren't attempted.
* Cloud services make it much harder to run your stack locally and on CI. There are solutions and workarounds, but they are all painful. And you always end up tied to the behaviour of the particular cloud services
> but you don’t need one tool to do all of those things
To get the same experience, you do. And I don't see why you would want multiple tools.
If anything, Kubernetes isn't nearly integrated and full-featured enough, because it has too many pluggable parts leading to too much choice and interfacing complexity. Like pluggable ingress, pluggable state database, pluggable networking stack, no simple "end to end app" solution ( KNative, etc), ... This overblown flexibility is what leads to most of the pain and perceived complexity, IMO.
Huh, I guess you are spot on. My first experience with kubernetes was k3s and I couldn't for a long time figure out what's all the fuss is about and where is all that complexity people talk so much about. But then I tried vanilla kubernetes.
Docker Swarm has all those features for example.
(Not that I am recommending Docker Swarm.)
That’s a joke.
That's just because AWS's Kubernetes offering is laughably bad.
There is huge difference in your experience whether you use Kubernetes via GKE (Autopilot) or any other solution (at least as long you don't have a dedicated infrastructure team).
It's just that GKE (Autopilot) does a lot of those out of the box for you that, so you get a much easier end-to-end experience for non-admins (= "request resources -> have them instantiated").
Kidding of course. If you need anything approximating Kubernetes, use it. If you just need one machine maybe don't.
... when the problem was lacking config management. They overshot; use the right tool for the job
But from an operational standpoint, when things are working, it usually behaves very well until you hit some rough edge cases (upgrades were much harder to achieve a couple of years back). But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.
I question the competence of anyone who does not question (and rag on) the prevalence of templating YAML.
> But rough edges exist everywhere, and when I get to a point where K8s hits a problem, I would think that it would be much worse if I wasn't using it.
Damn straight. It’s only bad because everything else is strictly worse.
So yaml formatters break it, humans struggle to generate code with proper indents, and it’s an insane mess. It’s horrendous.
What are the reasons to not use JSON rather than YAML? From my admittedly-shallow experience with k8s, I have yet to encounter a situation in which I couldn't use JSON. Does this issue only pop up once you start using Helm charts?
after using cdk i think that writing typescript to define infra is a significantly better experience
It ties you to k8s instead, and it ties you to a few company wide heroes, and that is not a 'benefit' as it's being touted here.
Being tied to a cloud is not a horrible situation either. I suspect "being tied to a cloud" is a boogeyman that k8s proponents would like to spread, but just like with k8s, with the right choices, cloud integration is a huge benefit.
> [...]
> * has a huge amount of pre-packaged software available (helm charts)
> * and most importantly: allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
NixOS. I have no clue about kubernetes, but I think NixOS even goes much deeper in these points (e.g. kubernetes is at the "application layer" and doesn't concern itself with declaratively managing the OS underneath, if I understand right). The other points seem much more situational, and if needed kubernetes might well be worth it. For something that could be a single server running a handful of services, NixOS is amazing.
Kubernetes manages a cluster, NixOS manages a single machine.
If the cluster is indeed necessary though, I think NixOS can be a great base to stand up a Kubernetes cluster on top of.
Disnix has been around for a long time, probably since before you ever heard of NixOS.
At the same time, people who suggest everyone to use kubernetes independently on the company maturity often forget how easy it is to run a service on a simple virtual machine.
In the multidimensional space that contains every software project, there is no hyperplane that separates when it’s worth to use kubernetes or not. It depends on the company, the employees, the culture, the business.
Of course there are general best practices, like for example if you’re just getting started with kubernetes, and already in the cloud, using a managed k8s service from your cloud provider could be a good idea. But again, even for this you’re going to find opposing views online.
This. I was trying to create some infrastructure and application once, using various AWS and off the shelf components. I stopped halfway through when I realized I was reinventing k8s, very poorly. That's when I switched gears and learned k8s.
With that said, I use it sparingly due to the inherent complexity it brings, but at least I have a better handle on how and when it should be used and when it should not, and what problems it solved, since I myself was trying to solve some of the problems.
* The declarative infra is EC2/ASG configurations plus Jenkins configurations * Client-side load balancing * ASG for autoscaling and recovery * Amazing observability with a home-grown monitoring system by 4 amazing engineers
Most of all, each of the above item was built and run by one or two people, except the observability stack with four. Oh, standing up a new region was truly a non-event. It just happened and as a member of the cloud platform team I couldn't even recall what I did for the project. It's not that Netflix's infra was better or worse than using k8s. I'm just amazed how happy I have been with an infra built more than 10 years ago, and how simple it was for end users. In that regard, I often question myself what I have missed in the whole movement of k8s platform engineering, other than people do need a robust solution to orchestrate containers.
Or at least that's how I got into k8s, because it allowed me to ship for 1/10th the price of my competitor.
And, for the record, observability is something very much unrelated to kubectl or k9s.
* declarative define your infrastructure
Declare in README.md that we have 3 web servers and that Bob Jones set them up and manages them. Include Bob's email address and phone number.
* gives you load balancing, automatic recovery and scaling
Load balancing via a load balancer or another scheme. DNS is good enough for some cases. There are other solutions.
Automatic recovery - daemon scripts on the box to start all services when the box boots. VPS provider bounces the box when it crashes. That's one. There are others.
Scaling - automatic scaling is not needed at vast majority of companies that are starting out. When we need to scale to 4 servers, change README.md and send Bob Jones a quick message.
* provides great observability into your whole stack (kubectl, k9s, ...)
This is a need that is introduced because of k8s. There's a lot less to observe without k8s, and tools exist for it.
It's like saying "my backhoe has great diagnostic tools for diagnosing backhoe issues." That's true, but I don't have a backhoe and don't need a backhoe for what I am doing.
* has a huge amount of pre-packaged software available (helm charts)
This is a need that is introduced because of k8s. See above.
* allows you to stand up mostly the same infrastructure in the cloud, on your own servers (k3s), and locally (KIND), and thus doesn't tie you into a specific cloud provider
> doesn't tie you into a specific cloud provider
Not a real problem for most companies. If you're preparing to change cloud providers from day one, you are likely spending time on the wrong problem.
> same infrastructure in different envs
This is a benefit, which other solutions come close to, but k8s shines at. You can go a long way without having reproducible multi-machine setups in different envs and can come pretty close when needed, with manual work.
> Kubernetes could have been much simpler, and probably was intentionally built to not be easy to use end to end.
If true, this is a strange design choice. I'd be wary of anything that was made complex just for the sake of it.
k8s gets enough flack without having to accuse it of being complex just for funsies.
> But it's still by far the best we've got.
k8s is the best we've got when we want k8s. The trick is to not want k8s for the sake of wanting k8s.
There are times when k8s provides tremendous value. Most companies who decide to use it do not have the problems that k8s promises to solve, and never will. Sadly, sometimes it's because they've spent their time and money on unnecessary complexity like k8s instead of building a product that delivers value.
Yes, k8s is complex. The tool matches the problem: complex. But having a standard is so much better than having a somewhat simpler undocumented chaos. “Kubectl explain X” is a thousand times better than even AWS documentation, which in turn was a game changer compared to that-one-whiteboard-above-Dave’s-desk. Standards are tricky, but worth the effort.
Personally I’m also very judicious with operators and CRDs - both can be somewhat hidden to beginners. However, the operator pattern is wonderful. Another amazing feature is ultra simple leader election - genuinely difficult outside of k8s, a 5 minute task inside. I agree with Paul’s take here tho of at least being extremely careful about which operators you introduce.
At any rate, yes k8s is more complex than your bash deploy script, of course it is. It’s also much more capable and works the same way as it did at all your developers previous jobs. Velocity is the name of the game!
So nearly a year later you end up writing the whole feature from scratch yourself.
I think a lot of the reaction here is a result of the age-old issues of "management is pushing software on me that I don't want" and people adopting it without knowing how to use it because it's considered a "best practice."
In other words, the reaction you probably have to an Oracle database is the same reaction that others have to Kubernetes (although Oracle databases are objectively crappy).
From service teams that have done the migrations, the things I hear consistently though are:
- when a Helm deploy fails, finding the reason why is a PITA (we run with --atomic so it'll rollback on a failed deploy. What failed? Was it bad code causing a pod to crash loop? Failed k8s resource create? who knows! have fun finding out!)
- they have to learn a whole new way of operating, particularly around in-the-moment scaling. A team today can go into the AWS Console at 4am during an incident and change the ASG scaling targets, but to do that with a service running in Kubernetes means making sure they have kubectl (and it's deps, for us that's aws-cli) installed and configured, AND remembering the `kubectl scale deployment X --replicas X` syntax.
[Both of those things are very much fixable]
This is different from waking people up at 4am frequently to bump up the number of replicas.
We've had pretty good success with simple HPAs.
I've been really happy with just using `envsubst` and environment variables to generate a manifest at deploy time. It's easy with most CI systems to "archive" the manifest, and it can then be easily read by a human or downloaded/applied manually for debugging with. Deploys are also just `cat k8s/${ENV}/deploy.yaml | envsubt > output.yaml && kubectl apply -f output.yaml`
I've also experimented with using terraform. It's actually been a good enough experience that I may go fully with terraform on a new project and see how it goes.
I cannot recommend terraform. I use it daily, and daily I wish I did not. I think Pulumi is the future. Not as battle tested, but terraform is a mountain of bugs anyway, so it can't possibly be worse.
Just one example where terraform sucks: You cannot both deploy a kubernetes cluster (say an EKS/AKS cluster) and then use kubernetes_manifest provider in a single workspace. You must do this across two separate terraform runs.
The problem is that they're custom and homegrown. Your organization alone invests in them, trains new staff in them, is responsible for debugging and fixing when they break, has to re-invest when they no longer do all the things you want. DIY frameworks ultimately end up as byzantine and labyrinthine as Kubernetes itself. The virtue of industry platforms like Kubernetes is, however complex and only half-baked they start, over time the entire industry trains on them, invests in them, refines and improves them. They benefit from a long-term economic virtuous cycle that DIY rarely if ever can. Even the longest, strongest, best-funded holdouts for bespoke languages, OSs, and frameworks—aerospace, finance, miltech—have largely come 'round to COTS first and foremost.
For example if you want to setup something complex like Ceph - Rook is a really nice way to do that. It's a very leaky abstraction so you aren't hiding all the complexity of Ceph but the declarative interface is generally a much nicer way to manage Ceph than a boatload of ansible scripts or generally what we had before. The key to understand is that helm or operators don't magically make infrastructure a managed 'turn-key' appliances, you do generally need to understand how the thing works.
But the article trending here a couple of days ago describes it well. https://www.theolognion.com/p/company-forgets-why-they-exist...
It is complex enough to make the k8s maintainers the heroes of the company. And this is where things tend to go sideways.
It has enough knobs and levers to distract the project from what they are actually trying to achieve.
Kubernetes is designed to solve big problems and if you don't have those problems, you're introducing a tonne of complexity for very little benefit. An ideal orchestrator would be more composable and not introduce more complexity than needed for the scale you're running at. I'd really like to see a modern alternative to K8S that learns from some of its mistakes.
You cannot pretend k8s doesn't exist in a k8s system.
If you are already using the cloud, maybe leverage abstraction already available in that context.
Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
None of that is prevented with SLA
Arguably if you can’t evaluate the raw cloud offerings and jump on a supposed silver bullet you need to stop immediately.
And with smaller companies I tend to find k8s way more cost effective. I pulled things I wouldn't be able to fit in a budget otherwise.
A few months later I transitioned the team to use containers with proper CI/CD and EKS with Terraform and Argo CD. The team and also the managers like it, since we could deploy quite quickly.
Our K8s clusters never goes more than a couple days without some sort of strange issue popping up. Arguably it could be because my company outsourced maintenance of it to an army of idiots. But K8s is a tool that is only as good as the operator, and competence can be hard to come by at some companies.
So I say unless you're at a company that pays top salaries for the top 5% of engineering talent, you're probably better off just using the AWS provided service.
Depending on your local market, AWS bills might be way worse than the cost of few bright ops people who will let you choose from offerings including running dev envs on random assortment of dedicated servers and local e-waste escapees
For larger scale orchestratiom, Hashicorp Nomad can also be a notable contender, while in some ways still being simpler than Kubernetes.
And even when it comes to Kubernetes, distros like K3s and tools like Portainer or Rancher can keep managing the cluster easy.
To be honest I hardly see any reasonable/actionable advice from Cloud/SAAS vendors. Either it is to sell their stuff or generic stuff like "One should be securing / monitoring their stuff running in prod". Oh wow, never thought or done any such thing before.
Choosing k8s is not just based on scaling requirements anymore. There are also benefits of being compatible with a rich ecosystem of software.
It's surely possible to cobble all of this together without k8s, but k8s' main advantage is exposing a standardized API that simplifies managing this entire ecosystem. It often makes it worth the additional overhead of adopting, understanding and managing k8s itself.
How many people out there really need C# or object oriented programming?
The argument you present might be valid if you decide to use a tech stack prior having much experience with it.
K8s is remarkably easier to retain institutional knowledge as well as spread it.
Also, in my experience, you either have to spend ridiculous amounts of money on SaaS/PaaS, or you find that you have to host a lot more than just your application and suddenly the deployment story becomes more complex.
Depending on where you are and how much you're willing to burn money, you might find out that k8s experts are cheaper than the money saved by not going PaaS.
Why would anyone expect it? It's not their job, is it? We don't expect backend devs to know frontend and vice-versa, or any of them to have AWS certification. Why would it be different with k8s?
> Just chucking a container on a non-k8s managed platform (e.g. Cloud Run) would be much simpler, and no pile of bash scripts.
Simpler to deploy, sure, but not to actually run it seriously in the long term. Though, if we are talking about A container (as in singular), k8s would indeed be some serious over-engineering
To add insult to injury, I've seen more than one use IaC cloud tooling as an install script vs a maintainable and idempotent solution. It's all quite sad really.
The former is a very small set involving having huge amounts of bare metal systems.
The latter is suprisingly large set of companies, sometimes even with one server.
For most use case k8s is not there to give you HA but to give you a standard way of deploying a stack, that being on the cloud or on prem.
What's wonderful is that when I work on multiple clouds, my knowledge transfers just fine. I don't think of the AWS solution or the GCS solution, I use the same kubectl to check out both, view logs, inspect and fix.
Even when I got tired of waiting for GKE to spin up a node, running Github actions on a self-hosted microk8s meant instant pod starts and very little fuss. But using Kubernetes meant I got to take advantage of the Github operator, which let me reuse the same machine for multiple builds without the headaches.
When I want to run some open source, I often find a helm chart the helps me get set up. Nowadays running open source packages can involve all kinds of dependencies, but getting it running on a k8s cluster to check it out, or even in prod, is a relatively straight forward editing of some values files. I've recently ran Uptrace and Superset that way. They're not a bajillion requests per second setups, they don't have to be, and it was far easier to set up than most methods.
I would say your friend is right. Few people _need_ k8s but it's one interface to a bunch of complicated proprietary stuff. I can know core small, core set of k8s tools really well and forget half of the junk that I ever knew about public clouds. It's all the same patterns, transferable and reliable.
Kubernetes might be the right tool for the job if we accept that this is a necessary evil. But maybe it's not? The idea that I might fail to collaborate with you because a third party failed because a fourth party failed kind of smells like a recipe for software that breaks all the time.
That's what I currently use Kubernetes for. What stack are you proposing instead?
If you need to replicate storage, share networks and otherwise share resources across multiple hosts, kubernetes is better suited.
But you'll also have much less control with compose, e.g. no limiting of egress/ingress and more.
I think this already introduces enough complexity and edge cases to make reinventing the wheel a bad idea. There's a lot involved in doing it robustly.
There are alternatives to Kubernetes (I prefer ECS/Fargate if you're on AWS), but trying to do it yourself to a production-ready standard sets you up for a lot of unnecessary yak shaving imho.
k8s is a large and complex tool. Anyone who's run it in production at scale has had to deal with at least one severe outage caused by it.
It's an appropriate choice when you have a team of k8s operators full-time to manage it. It's not necessarily an appropriate choice when you want a zero-downtime deploy.
Are you talking about a full self-run type of scenario where you setup and administer k8s entirely yourself, or a managed system or semi-managed (like OpenShift)? Because if the former then I would agree with you, although I wouldn't recommend a full self-run unless you were a big enough corp to have said team. But if you're talking about even a managed service, I would have to disagree. I've been running for years on a managed service (as the only k8s admin) and have never had a severe outage caused by K8s
There are a lot of reverse proxies that will do this. Traditionally this was the job of a load balancer. With that being done by "software" you get the fun job of setting it up!
The hard part is doing it the first time, and having a sane strategy. What you want to do is identify and segment a portion of your traffic. Mostly this means injecting a cooking into the segmented traffics HTTP(S) requests. If you dont have a group of users consistently on the new service you get some odd behavior.
The deployment part is easy. Cause your running things concurrently then ports matter. Just have the alternate version deployed on a different port. This is not a big deal and is supper easy to do. In fact your deployments are probably set up to swap ports anyway. So all your doing is not committing to a final step in that process.
But... what if it is a service to service call inside your network. That too should be easy. Your passing id's around between calls for tracing right? Rather than "random cookie" you're just going to route based on these. Again easy to do in a reverse proxy, easier in a load balancer.
It's not like easy blue green deploys are some magic of kuberneties. We have been doing them for a long time. They were easy to do once set up (and highly scripted as a possible path for any normal deployment).
Kubernetes is to operations what rails is to programing... Its good, fast, helpful... till it isnt and then your left having buyers remorse.
LOL so, rather than linting YAML, bring in a whole programming language runtime plus third party library, adding yet another vendor lock, having to maintain versions, project compiling, moving away from K8S, adding mental overhead...
I have drastically reduced the amount of errors, mistakes, bugs, plain old wtf-induced hair pulling, by just mandating avoidance of YAML (and Helm) and using Jsonnet. Sure, there was some up-front work to write library code, but afterwards? I had people introduced to JSonnet with example deployment on one day, and shipping production-ready deployment for another app the next day.
Something we couldn't get with yaml.
I was talking to someone from a local startup a couple weeks ago who was trying to explain their devops stack. The number of different tools and platforms they were using was in the range of 50 different things, and they were asking for advice about how to integrate yet another thing to solve yet another self-inflicted problem.
It was as though they forgot what the goal was and started trying to collect as much experience with as many different tools as they could.
hopefully jsonnet or that apple thing will get more traction and popularlity.
Our team is happy to write raw YAML and use kustomize, because we prefer keeping the config plain and obvious, but we otherwise pretty much follow everything here.
Managing k8s, for me at least, is a lot easier than juggling multiple servers with potentially different hardware, software, or whatever else. It’s rare that businesses will have machines that are all identical. Trying to keep adding machines to a pool that you manage manually and keep them running can be very messy and get out of control if you’re not on top of it.
k8s can also get out of control though it’s also easier to reason about and understand in this context. Eg you have eight machines of varying specs but all they really have installed is what’s required to run k8s, so you haven’t got as much divergence there. You can then use k8s to schedule work across them or ask questions about the machines.
k8s's worst property is that it's a cleverness trap. You can do anything in k8s whether it's sane to do so or not. The biggest guardrail against falling into is managing your k8s with terraform-ish so that you don't find yourself in a spot where "effort to do it right" >> "effort to hack in YAML" and finding your k8s cluster becoming spaghetti.
Re: cleverness trap. I feel like this is the tragedy of software development. We like to be seen as clever. We are doing "hard" things. I have way more respect for engineers that do "simple" things that just work using boring tech and factor in whole lifecycle of the product.
Not everyone has money to burn, even back in ZIRP era.
And before you trot out wages for experienced operations team - I've regularly dealt with it being cheaper to pay for one or two very experienced people than deal with AWS bill.
For the very simple reason that cloud provider's prices are scaled to US market and not everyone has US money levels.
I wouldn't start with k8s and instead opt for ASGs until you reach the point where you look at your AWS account and see a bunch of EC2 instances sitting underutilized.
If there were an alternative to Kubernetes that were just 10% less confusing, complicated, opaque, monolithic, clunky, etc, we would all be using it. But because Kubernetes exists, and everyone is using it, there's no point in trying to make an alternative. It would take years to reach feature parity, and until you do, you can't really switch away. It's like you're driving an 18-wheeler, and you think it kinda sucks, but you can't just buy and then drive a completely different 18 wheeler for only a couple of your deliveries.
You probably will end up using K8s at some point in the next 20 years. There's not really an alternative that makes sense. As much as it sucks, and as much as it makes some things both more complicated and harder, if you actually need everything it provides, it makes no sense to DIY, and there is no equivalent solution.
And often pushed Nomad to this day surprises me with randomly missing a feature or two that turns out to be impactful enough to want to deal with more complexity because ultimately the result was less complexity in total.
People make big claims but then it's not declarative enough to look up a resource or build a dependency tree and then your context deadline is exceeded.
As someone who is always working "under" a particular set of infrastructure choices I want people who write this kind of article to understand something: the people who dislike particular infrastructure systems are by-in-large those who are working under sub-optimal uses of them. No one who has the space to think about "if their infrastructure choices will create an effect" in the future hates any infrastructure system. Their life is good. They can choose and most everyone agrees that any system can be done well.
The haters come from being in situations where a system has not been done well - where for whatever combination of reasons they are stuck using a system that's the wrong mix of complex / monitorable / fragile / etc. It's true enough that, if that system had been built with more attention to its needs, that people would not hate it - but that's just not how people come to hate k8s (or any other tool).
* built a jar (it was a jvm shop)
* put it on a docker image
* put that on an ami
* then had a regular aws load balancer that just combined the ami with the correctly specced (for each service) ec2 instances to cope with load
it was SIMPLE + it meant we could super easily spin up the previous version ami + ec2s in case of any issues on deploys (in fact, when deploying, we could keep the previous ones running and just repoint the load balancer to them)
ps putting the jar on a docker image was arguably unnecessary, we did it mostly to avoid "it works on my machine" style problems
Is there a clear example of this? E.g. is kubernetes inherently unable to start a pod (assuming the same sequence of events, e.g. warm/cold image with streaming enabled) under 500ms, 1s etc?
I am asking this as someone who spent quite a bit of time and wasn't able to bring it down 2s< mark, which eventually led us to rewrite the latency sensitive parts to use Nomad. But we are currently in a state where we are re-considering kubernetes for its auxilary tooling benefits and would love to learn more if anyone had experiences with starting and stopping thousands of pods with the lowest possible latencies without caring for utilization or placement but just observable boot latencies.
a) preload all images of course
b) there's enough of nodes with enough capacity
c) the pods don't use anything that has possible longer latency (high latency CSI etc.)
d) you might want to write custom scheduler for your workloads (it could take into account what images are preloaded where, etc)
Big possible win is custom scheduling, but barely anyone seems to know it exists
(I made up the dozen number, but my point is that that design would be perfectly acceptable given Kubernetes' design constraints)
Something like Knative can allow for faster startup times if you follow the common best-practices (pre-fetching images, etc.), but I'm not sure if it supports enough of the session-related feature that you were probably looking for to be a stand-in for Plane.
k3s really hammers home the "kubernetes is a set of behaviors, not a set of tools" stuff when you realize you can ditch etcd entirely and use sqlite if you really want to, and is a good learning environment.
If you want something crazy all-in-one for homelab check out https://github.com/azukaar/Cosmos-Server
For something more production oriented https://github.com/basecamp/kamal
- Containers
- Pods
- Deployments
- Services
- Ingresses
There are certainly other concepts you can learn, but you aren't often dealing with them (just like you aren't dealing with them when working with something like docker compose).
Haven't had an issue once I fixed sizing.
I'd say that if you aren't big enough to have dedicated SRE then k8s is not for you. However, it really only takes 1 or 2 people to manage pretty large clusters with 100s or 1000s of deployments.
That's where my company currently finds itself.
We have jobs that users initiate that use 80+GB of memory and a few dozen cores. We run only one pod per node because the next size up EC2 costs a fortune and performance tops out on our current size.
These jobs are triggered via a button click that trigger a lambda that submits a job to the cluster. If it is a fresh node, user has to wait for the 1gb container to download from ECR. But it is the same container that the automated jobs that kick off every few minutes also uses, to rarely is there any waiting. But sometimes there is.
Should we be running some sort of clustering job scheduler that gets the job request and distributes work amongst long running pods in the cluster? My fear is that we just creat another layer of complexity and still end up waiting for the EC2, waiting for the pod to download, waiting for the agent now running on this pod to join the work distribution cluster.
However, we probably could be more proactive with this because we could spin up an extra pod+EC2 when the work cluster is 1:1 job:ec2.
Thoughts?
We're in the process of moving to Karpenter, so all this may be solved for us very soon with some clever configuration.
For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway.
What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency.
In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.
This is the key point. Even getting to the point where I could install Kubernetes myself on my own hardware took weeks, just understanding what hardware was needed and which of the (far too many) different installers I had to use.
That being said, helm templates are disgusting and I absolutely hate how easily developers complicate their charts. Even the default empty chart has helpers. Why, on Earth, why?
I almost fully relate to OPs aproach to k8s but I think with their simplified approach helm (the current one) could work quite well.
At most places, your cluster configuration is probably pretty set in stone and doesn't vary a ton.
We are here because we wanted a way to deal with software that each need different versions of libraries or configuration, so we invented containers. Then we wanted to run multiple services, so we invented orchestrators. But now applications are deployed with cluster-scoped operators that depend on cluster-scoped definitions, and we need to run separate services in separate cluster. It's like with need to create containerization for Kubernetes apps again.
The only thing I learned is about Caddy as a cert-manager replacement, even though I have used, extended and been pretty happy with cert-manager. The rest is hard to read ;(.
Reading about JamSocket and what it does, it seems that it essentially lets you run Docker instances inside the Jamsocket infrastructure.
Why not just take Caddy in a clustered configuration, add some modules to control Docker startup/shutdown and reduce your services usage by 50%? As one example.
The earliest version of the product really was just nginx behind some containers, but we outgrew the functionality of existing proxies pretty quickly. See e.g. keys (https://plane.dev/developing/keys) which would not be possible with clustered Caddy alone.
I did add the caveat of "with modules" and the idea of sharing values around to different servers would be easy to do, since you have Postgres around as a database to hold those values/statuses.
The next one was Microservices. Everyone was doing something with microservices and I was just on a good 'ole Ruby on Rails monolith. Again, the HN stories came and went "Why we broke down our simple CRUD app into 534 microservices".
The final one was Kubernetes. I was a Cloud consultant in my past life and had to work with a lot of my peers who had the freedom to deploy In any architecture they saw fit. A bunch of them were on Kubernetes and I was just on a standard Compute VM for my clients.
We had a requirement from our management that all of us had to take some certification courses so they would be easily to pitch to clients. So, I prepped for one and read about Kubernetes and tried deploying a bunch of applications only to realize it was a very complex piece of moving parts - unnecessarily I may add. I was never able to understand why this was pushed on as normal. It made my decision to not use it only stronger.
Over the course of the 5 year journey, my peers' apps would randomly fail and they would be sometimes pulled over the weekends to push fixes to avert the P1 situation whilst I would be casually chilling in a bar with my friends. My compute engine VM, till date, to its credit has only had one P1 situation yet. And that was because the client forgot to renew their domain name.
Out of all the 3 hype cycles that I avoided in my career, the Kubernetes is the one I really am thankful of evading the most. This sort of complexity should not be normalised. I know this maybe unpopular opinion on HN, but I am willing to bite the bullet and save my time and my clients' money. So, thanks for the hater's guide. But, I prefer to remain one. I'd rather call a spade one.
Some time down the road we got acquired, and the company that acquired us ran their services in their own Kubernetes cluster.
When we were talking with their two person devops team about our architecture, I explained that we deployed some of our services on ECS. "Have you ever used it?" I asked them.
"No, thank goodness" one of them said jokingly.
By this time it was clear that Kubernetes had won and AWS was planning its managed Kubernetes offering. I assumed that after I became familiar with Kubernetes I'd feel the same way.
After a few months though it became clear that all these guys did was babysit their Kubernetes cluster. Upgrading it was a routine chore and every crisis they faced was related to some problem with the cluster.
Meanwhile our ECS deploys continued to be relatively hassle free. We didn't even have a devops team.
I grew to understand that managing Kubernetes was fun for them, despite the fact that it was overkill for their situation. They had architected for scale that didn't exist.
I felt much better about having chosen a technology that didn't "win".
What are the best resources to learn simple k8 in 2024?
Sure, everyone has their own product and experience and it's fine to express it, but I don't get the usage of other decisions such as "no to services meshes", "no to helm" and many more.
You don't want to ideally reinvent the wheel for every workload you need (say you need a OIDC endpoint, an existing application): you are tempted to write everything from scratch by yourself, which is also fine, but the point is: why?
Many products deliver their own Helm package. And if you are sick of writing YAML, I would look for Terraform over Pulumi, for the reason that you use the same tool for bringing up Infrastructure and then workloads.
Kubernetes itself isn't easy to be used, in many cases you don't need it, but it might bring you nice things straight out of the box with less pain than other tooling (e.g. zero downtime deployments)
They do text-replacement templating for YAML.
I have once spent a month, being quite experienced k8s wrangler, trying to figure out why Helm 2 was timing out, only to finally trace it down to how sometimes we would get wrong number of spaces in some lines.
Service meshes are a different kettle of fish. hey add exciting new points of failure where they need not exist, and while there are definitely use cases for them, I'd default to avoiding them until somebody proves the need for one.
Allow k8s disallow any service meshes.
The whole thing is a bolt-on that has to spend a ton of time working around the limitations of the foundation, and it shows.
Unfortunately there seems to be zero interest in fixing that and so much sunk cost in existing Unix/Posix designs that it seems like we are completely stuck with a basic foundation of outdated brittleness.
What I think we need:
* An OS that runs hardware-independent code (WASM?) natively and permits things like hot updates, state saving and restoration, etc. Abstract away the hardware.
* Native built-in support for clustering, hot backups, live process migration between nodes, and generally treating hardware as a pure commodity in a RAIN (redundant array of inexpensive nodes) configuration.
* A modern I/O API. Posix I/O APIs are awful. They could be supported for backward compatibility via a compatibility library.
* Native built-in support for distributed clustered storage with high availability. Basically a low or zero config equivalent of Ceph or similar built into the OS as a first class citizen.
* Immutable OS that installs almost instantly on hardware, can be provisioned entirely with code, and where apps/services can be added and removed with no "OS rot." The concept of installing software "on" the OS needs to be killed with fire.
* Shared distributed network stack where multiple machines can have the same virtual network interfaces, IPs, and open TCP connections can migrate. Built-in load balancing.
I'm sure people around here can think of more ideas that belong in this list. These are not fringe things that are impossible to build.
Basically you should have an immutable image OS that turns many boxes into one box and you don't have to think about it. Storage is automatically clustered. Processes automatically restart or, if a hardware fault is detected in time, automatically migrate.
There were efforts to build such things (Mosix, Plan 9, etc.) but they were bulldozed by the viral spread of free Unix-like OSes that were "good enough."
Edit:
That being said, I'm not saying Kubernetes is good software either. The core engine is actually decent and as the OP said has a lot of complexity that's needed to support what it does. The ugly nasty disgusting parts are the config interface, clunky shit like YAML, and how generally arcane and unapproachable and ugly the thing is to actually use.
I just loathe software like this. I feel the same way about Postgres and Systemd. "Algorithmically" they are fine, but the interface and the way you use them is arcane and makes me feel like I'm using a 70s mainframe on a green VT220 monitor.
Either these things are designed by the sorts of "hackers" who like complexity and arcane-ness, or they're hacks that went viral and matured into global infrastructure without planning. I think it's a mix of both... though in the case of Postgres it's also that the project is legitimately old. It feels like old-school Unix clunkware because it is.
[Edit] Parent comment is almost entirety completely different after that edit to what I responded to. But I think my point still stands. One day, hopefully in my lifetime, we shall see it.
The problem is that right now this gets you lock-in to a proprietary cloud. There are some loose standards but the devil's in the details and once you are deployed somewhere it's damn hard to impossible to move without serious downtime and fixing.
I can’t say I know it myself. It always looks good on paper. Strangely nobody uses it. There must be a catch that detracts from it?
Hopefully we should get OpenFaas and Lambda, in the same way we have ECS and EKS. Standardised ways to complete tasks, rather than managing imaginary servers.
We are still early in the cycle.
If CoreOS actually wanted to make distributed computing easier, they'd make patches for the Linux kernel (or make an entirely different kernel). See the many distributed OS kernels that were made over 20 years ago. But that's a lot of work. So instead they tried to go the cheap and easy route. But the cheap and easy route ends up being much shittier.
There's no commercial advantage to building a distributed OS, which is why no distributed OS is successful today. You would need a crazy person to work for 10 years on a pet project until it's feature-complete, and then all of a sudden everyone would want to use it. But until it's complete, nobody would use it, and nobody would spend time developing it. Even once it's created, if it's not popular, still nobody will use it (you can use Plan9 today, but nobody does).
Maybe not, but I'm confident that the system you're describing is impossible to build in a way that is both general and efficient.
If I have a block of storage that boots on one system, it should boot on another. An OS should poll for available services, allow them to define APIs and be utilized, but no bloat should be added to a base to support elective services.
Everything should be microkernel too, but now I’m just venting.
Some technologies (like Kubernetes) tend to attract discussions where half of the commenters completely ignore the original article, so we end up having a weekly thread about Kubernetes where the points of the article (which are interesting) are not able to be discussed because they are drowned out by the same unstructured OT discussions?
At the time of this posting there are ~20 comments with ~2 actually having anything to do with the points of the article rather than Kubernetes in general.
Discussions of k8s pitfalls and successes in general seems to be very much in line with what the article is advocating. And, to that point, there's frankly just not a whole lot interesting in this article for discussion "We avoid yaml and operators"... Neat.
Yeah, and I think that provides a good basis to discussion, where people can critique/discuss whether the evaluation that the author has made are correct (which a few comments are doing). At the same time a lot of that discussion is being displaced by what I would roughly characterize as "general technology flaming" which isn't going anywhere productive.
I am rather happy that people are having general purpose discussion about K8s.