Kubernetes reinvented virtual machines in a good sense
iximiuz.com
iximiuz.com
Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
It seems rather that you don't know what kubernetes is. His comment is justified. You can put your pitchfork back.
> Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
It seems that you don't understand ehat he's trying to say.
The company, for which I helped migrating to k8s, pays me 6 figures to keep maintaining it, thinks otherwise (also look at my username ;) ). All of our developers love it because it replaced some weird hacked together deployment stuff.
Which part do you think he is right or wrong?
It just seems that in every thread people moan about k8s. But it seems that it's usually people who are overwhelmed by it, which is a legitimate reason not to use it. But just saying that it's generally bad is what starting to annoy me a bit.
If you’re being paid well, obviously you’ll be annoyed by concepts contrary to your work.
e: edits are hated, but the downvotes prove out: Kubernetes is simply OpenStack for hipsters.
- Upton Sinclair.
I’m currently helping my employer switch everything to kubernetes and we’re replacing a lot of really, truly over-engineered trash.
I will say I do hate yaml but there aren’t a ton of choices, tbh.
I have spent my entire Kubernetes career unfucking the people three degrees removed from you who hear advocacy like this and say wow, yeah, the people with gripes about Kubernetes are just moaning and overwhelmed. Meanwhile, a team at an unnamed government agency just allowed Accenture to con them into turning multiplayer Excel, a process which used to take two humans about a week, into an $18 million annual AWS system funded by the U.S. taxpayer. Because of choices-du-jour, including Spark and Kubernetes.
It just seems that in every thread people moan about people moaning about Kubernetes and then project their interpretation of what that moaning is after starting from the presupposition that it's moaning. If you'd instead listen to the other side instead of the basically-crypto-advocacy-train-choo-choo of "if you have gripes, you just don't get it," you'd understand the cons a little better. Such as my entire decade of professional Kubernetes consulting being every single computing project, including in the Fortune 500, the Department of Defense, and at second-tier VC startups in the Bay Area, being a complete operational disaster that's bleeding capital like it's free.
Capital is no longer free. Read the news.
I now wield my certifications to advise against Kubernetes in any capacity because it is a strategic risk unless resourced exactly right and funded exactly right (which is impossible to justify on infra budgets), simply because of the experience I have watching people like you come in, explain to leadership that it's fine, and then ride that salary until it doesn't suit you anymore and leave a system that takes 5-6 FTEs to comfortably administer on whoever gets to pick up the ball. For every comment you see on Hacker News where someone like you is saying "yeah, Kubernetes runs great for us," they're (a) leaving out the operational toil, particularly in the break-fix department where the entire Unix paradigm changes and (b) not realizing that there are five other teams like them that are completely underwater while trying to put out their overengineered tire fire.
And remember, nearly every project using Kubernetes is trying to do multiplayer Excel. That's it. And the more we invest in trying to make the world's most overcomplex computing system do that use case even harder, the less we think about "isn't there a simpler way to do all of this from first principles?"
GP is absolutely right that it is insane overkill with no conceivable way to pay back the complexity investment in all but a few niche cases. Working at a FAANG isn’t the only niche where full-Kube makes sense, there are others, but it’s a niche thing, with the niche being roughly: I rack and run physical gear on a limited number of SKUs at “cattle not pets” scale.
If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust than any half-assed Borg-alike setup most people would come up with.
They’re happy to take your money if you yo dawg like imaginary computers so I put a docker in your Xen VM so you can Kubernetes while you pay me big bucks, but that’s because they’re not running charities.
What commodity tool should I be replacing it with that gives me:
* super for heterogenous workloads (web servers, big batch loads, streaming workloads)
* autoscaling the workloads
* autoscaling the cluster
* auto-restarting workloads
* basically automatic networking
* the ability to have all that without the need to setup separate infrastructure, and to have the orchestrator move workloads around to optimise machine use
* the ability to hire people with experience in it already
* ability to run stateful workloads, and have disks managed and moved for you
* the ability to do all that by just providing a config
* the ability to do all that without having to let people have actual access to the machines/OS
* per container/workload topology constraints and requirements, that are network and data centre aware, without having to write any code
* have access to a wide range of tooling and docs
* be able to drop in workloads that automatically handle log collection, etc. without needing to configure each deployment.
* ability to swap out Network ingress/reverse proxy without having to redeploy or reconfigure your applications.
* ability to drop in any number of tools to do traffic shadowing, mirroring, splits, etc without reconfiguring or redeploying your applications
* ability to do all this without your devs needing to be networking experts.
* not locked to a specific cloud provider
I’m sure there’s tools out there that cover different subsets of those features, there’s probably tools that do all of them and more. K8s isn’t perfect, but it gives you an awful lot for some reasonable tradeoffs. Sure, if you’re deploying a single web-server/app, you shouldn’t bother with all this; and data centre level stuff probably has tools that make these features look like a cute picnic, but many devs I’ve worked with haven’t the capacity, ability or desire to figure that out.
> If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust
Sure, but you’ve still got to figure out how you want to setup, monitor, configure all of that, then stitch it together and maintain it, and also not get in the way of your main application development. To me, that’s what K8s gives me: I rely on the cloud provider to give me good machines, good networking, good storage primitives, and to heal/replace them when they fail, and K8s gives me a way to use all of that nicely, without needing to build and maintain it myself, and without my applications needing to know everything.
I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases.
Learning how to package services and have them run anywhere? A lost art.
Cloud agnosticism is a red herring. No matter what you do today you are not cloud agnostic.
Nobody ever said it wasn’t possible. Nobody is saying “you can’t run things the way you want”, the argument is “you keep saying K8s is bad, here’s some reasons why it’s maybe not”.
> I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases.
If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert.
> Learning how to package services and have them run anywhere? A lost art.
The issue isn’t my teams code-if we were only running our own stuff, and we only used a single language, packaging would likely be straightforward. But it’s the fact that we run several other applications, of varying degrees of “packaged” and the fact that I can barely get the Python devs to use a virtualenv properly that makes this statement a bit too unreasonable in my experience. Containers may be messy, but at least I can run 3 different spark pipelines and 4 Python applications without needing to dig into the specifics of “how this particular app packages it’s dependencies” because that’s always an awful experience.
Let's define what you're talking about here and I can show you the way.
What is the difficulty in autoscalling Spark jobs? What jupyter env do they use? You say grpc api. That's genericm what do those services really do? Sync request/response backed by a db? Async processing? What infra do they need to "work"
Where are you running your workloads? Cloud? Which one?
I mean, running spark is an awful experience at the best of times, but let’s just go from there.
Spark drivers pull messages off Kafka, and scale executors up or down dynamically based off how much load/messages they have coming through. This means you need stable host names, ideally without manual intervention. The drivers and the executors should also use workload-specific roles - we use IRSA for this, and it works quite nicely. Multiple spark clusters will need to run, be oblivious to each other, and shouldn’t “tread on each other” so the provisioning topology should avoid scheduling containers from the same job (or competing jobs) onto the same node. Similarly, a given cluster should ideally be within an AZ to minimise latency, doesn’t matter which one, but it shouldn’t be hardcoded, because the scheduler (I.e. K8s) should be able to choose based on available compute. Some of the jobs load models, so they need an EBS attached as scratch space.
> What jupyter env do they use?
We use jupyterhub wired up to our identity provider. So an analyst logs on, a pod is scheduled somewhere in the cluster with available compute, and their unique EBS volume is automounted. If they go to lunch, or idle, state is saved, and the pod is scaled down and the EBS volume is auto-unattached.
> That's genericm what do those services really do? Sync request/response backed by a db? Async processing? What infra do they need to "work"
The API stuff is by far the easiest. Request response, backed by DB’s, the odd analytics tool and monitoring tool as well. Servers themselves autoscale horizontally, some of them use other services hosted within the same cluster, all east-west traffic within the cluster is secured with mTLS via linkerd, between that and our automatic metric collection, we get automatic collection of metrics, latencies, etc. like what you get with the AWS dashboards, but more detail (memory usage for one), automatic log collection too.
> Where are you running your workloads?
AWS, but minimal contact - S3 is basically a commodity API nowadays, the only tight integration is IRSA, which I believe GCP has a very similar version of. So most of this should work in any of the other clouds, with minimal alteration.
a lot of scripts crammed together that could be broken by any single dev
Have you ever done IaC where the whole copy of the infra came up on deployment and traffic was shifted between old and new? And developers could spin up their own stack with one Cloudformation or Terraform command? Have you used cloud primitives that solve most of the problems you are reinventing with k8s?
By the way, Terraform and k8s are not exclusive, I use them together.
I can tell you how we managed to do anything.
We had few servers. They run docker-composes scattered all over people's home directories.
We had one nginx entrypoint. It's forbidden to restart it at work hours because it won't restart and there will be hours outage. You expected to restart it at weekend and then spend hours trying to start it.
Some docker images are built locally.
Backups are basically not worked before I came to this company. It was the first thing I fixed. Now we can restore our system if server dies. It'll take a week or two.
Kubernetes is going to be sane exit.
Yes, I know that it's possible to build sane environment. Problem is people don't bother to do that. They build it in the worse possible way. And with kubernetes, worst possible way is much better because kubernetes provides plenty of quality moving blocks which you don't need to reinvent. You don't need to reinvent nginx ingress with all the necessary scripts to regenerate its config, because you've got ingress-nginx. You don't need to invent certbot integration because you've got cert-manager (which is my personal favourite piece of k8s). You don't need to invent your docker-composes storage conventions, user permissions (777 everywhere, right), because k8s got you covered. All you need is to learn its conventions and follow them.
AWS.
I would have agreed with you three years ago, but today there's a ton of surprisingly good off the shelf solutions, you can get very, very far in a brief amount of time with a single engineer. A ton of the functionality exposed by Kubernetes is absolutely necessary at any scale and very difficult to do yourself. "Don't let any process on this host talk to this port except these ones" is something that is so much easier in K8s than anything else I've ever encountered.
One of the scenarios I’ve seen Kubernetes replace is something like tying together Packer, Chef, Jenkins, Terraform, And Vault to get an EC2 instance propped up, but that’s just to deploy the code.
What was particularly annoying to me was that the developers were already using containers to build their code and another container for unit testing. So switching to Kubernetes would’ve been far easier. I had to go in and deconstruct the work they were doing with containers to then get their app deployed to an EC2 instance.
Although at this level you’re still not saving a ton by going the Kubernetes route. You get a ton of benefits with it, though. Even just tiny things like being able to get the config from a deployment using kubectl is so good. And then you can modify it, too! That’s been super helpful for troubleshooting.
My point was that unless you’re running on metal, the cloud people built all this shit, and much, much better than k8s (or Tupperware for that matter).
They’ll stick you Docker container inside a VM happily because people pay them to do it, but it’s redundant.
Google didn’t OSS Tensorflow out of the goodness of their hearts, they did it to sell TPU hours on GCP.
k8s is likewise not charitable giving: it’s to sell managed Kubernetes on GCP, and subtle nudges to get people to make it their careers are uh, just business I guess.
Provisioning as code is in a sad state: Chef/Ansible/Salt/friends can’t deliver on idempotency, Nix is too complicated for most. So people use a container primitive (Docker) to work around that.
This has nothing, absolutely nothing, to do with service discover and failover and auto-grow and other Kube shit.
AWS does all that already. Better.
This is a borderline-cynical business strategy on the part of the cloud providers, and judging from the religious fervor around k8s (someone upthread has it in their username for Christ’s sake), it’s working.
This rings true every time I see someone talking about K8s. Companies just want something that is customizable to their workflow, they don't want a ready-made PaaS offering. Google's play was to provide an industry standard API interface that is available everywhere (which is essentially what Kubernetes is). Now that the control plane for compute is commoditized, GCP et al can sell complements - BigQuery, TensorFlow, GCS etc.
I think Anthos-like solutions are the future. Just buy the cheapest hardware from your local Datacentre or BareMetal providers like Hetzner and run GCP, AWS or some other flavor of software on top of it. This is also where Teleco Clouds are moving with OpenRAN and 5G, they are tired of being locked-in to full stack vendors like Nokia and Ericsson and want interchangeable solutions compatible through a common interface
Well done sir or madame.
Somewhere between "I'm small enough that all of the cloud providers are about the same" and "I'm big enough that I can leverage them against each other to get a better deal, but still small enough to not be on metal" there is maybe a sweet spot where I really do need to be mobile across providers in a push-button way. People in e.g. blockchain use k8s to good effect because they want to run a big distributed system with loose-ish central organization, and that's a real thing. It's also a niche.
I never said Kubernetes was useless full stop. I said that places where it's the right call are niche.
My current gig is humble, a usually-raman-profitable type garage band, we're like 5 engineers with Ethernet cables in our living rooms and a bigger AWS bill than feels fair, and if I proposed to any of the ex-FAANG type people involved that we should get serious about Kubernetes they'd laugh themselves pink in the face and resume typing. We can give our machines names and remember them all, there are a few dozen of them, but that's still "pets".
If we're fortunate and hard-working enough to need 1k machines, I'll spin up a ZK instance or something and write a little Nix/bash.
I won't be networking at KubeCon 2023 unless I lose a bet to one of my peers. I've got work to do.
I do entirely agree with you, I think: there are too many companies using Kubernetes prematurely, but...
It's kind of turned out okay? Because of this, the ecosystem seems to be improving and it's easier than ever for small shops to get started. And selling software that targets Kubernetes is leagues better than targeting Ubuntu or Red Hat or specific kernels.
> I'll spin up a ZK instance
If I never have to run another ZooKeeper instance again I'll die happy.
It's a powerful abstraction and it's very liberating.
I understand the cynicism, don't get me wrong, but that doesn't mean it's all bad. It definitely feels like a net positive to society even if at times it's unbearable.
I'm going to try single-shot learning on one of the newer language models with "I can confirm your viewpoint. I'm single engineer building self-managed k8s for my company." because that remark, reasonable or not, so neatly encapsulates the current zeitgeist.
If you're trolling, respect. And if you're not, my 2c: flash an AMI and work on your product.
It's not the machines. It's the comosible components, the service building blocks I can get with an API call.
* Want traffic into a cluster, ask for an Ingress. * Want to auto-scale? Ask for a HorizontalPodAutoscaler. * Want to manage CPU and memory rather than number of containers? VerticalPodAutoscaler. * Want to point your monitoring system at a port? PodMonitor or ServiceMonitor
And of course, you can extend on the basic plugins. Want a bunch of databases? Maybe install the Vitess operator and simply make an API call to create an instance of VitessCluster. There are a dozen options, depending on what flavor of database you want.
Oh, and you don't need a sysadmin/SRE to do all this for you. You can be granted rights to a Namespace and compose everything you need in your namespace.
Saying that you should use it because people know does not hold water. Also, anecdotally very few people understand k8s (setup and running it). Most people just know how to deploy apps on it (and even there most impressionable are young devs that haven't done things in a preK8s world)
It’s so much more than it’s headline.
What is a docker container? As a matter of fact: what is a container?
A container is not a replacement for properly securing/isolating your app. Any flaw in any of the mechanism the kernel uses to provide the "containerization" features can end up with all the containers on the node compromized. Also, unless you're really disciplined about how you're building your images you probably already have a bunch of zero days lurking in your deployments.
It's all fun and games until we actually need accountability and the buck has to stop with someone.
I don't think this is that bad tbh. I runs my unit tests on container construction and while it was tedious to develop, mostly because i cannot run docker on my work PC, i understand that it makes it easier to inventory and check the advancement. The management will always love it, so i think kubernetes is here to stay. Not because it solve a lot of engineering issues (not for me anyway, i'd rather run a cron in my VM than declare a kube cronJob), but because it solve a lot of other fondamental issues for the management (inventory, visibility on existing/running projects. And everybody use the same system without using the _same_ system.)
Have a look at tilt.dev You can get it to build in-cluster too so you only need the CLI. I've really enjoyed its workflow.
Or try podman-remote. Should be easier to get those CLI tools on your work PC.
Like i said, it is fine, i'm not mad about it, it took some time to understand but i get it now and while still less efficient, i'm way better at it, and when i finish this project the next one will be easier. Also, it is important that the whole team use the same product: it was a good reason to loose one man-month to get me up to date with the rest of the team and facilitate my next "app".
Also I noticed that once developers "get it", they actually like it a lot.
And I think k8s is here to stay. If what you have is a bunch of containers you can easily push that into k8s. It's not any more complex than any other solution you would use to deploy them.
I really like k8s architecture. It's just miracle architecture that I'd never come with myself. It's very extendable. I'm very inspired by it and I hope to apply it to my own projects some time.
But UI is terrible.
scopeSelector:
matchExpressions:
- scopeName: PriorityClass
operator: In
values:
- middle
Also I'm not sure that I like all this modular nature of k8s. Sometimes it makes sense, for example for CSI. But why should I choose between CNI implementations? I just want best one and that's about it. I want it to be from Google. I don't need to choose between kube-apiserver implementations or kube-scheduler implementations, thanks god.
Also the whole kubeadm story seems half-baked. It works, but far from ideal. I guess it'll get better with time.
But those are small complains. Like I think that it has a good potential to improve. But there's nothing to replace it. Even if someone will build some lambda swarm which is better than k8s, nobody's going to migrate to it, because k8s is offered by so many cloud providers which made investments and are not going to move to other solutions in the foreseeable time.
Honestly, it is. You're supposed to write your own operator that would in turn generate all this boilerplate properly — you can look at a tech blog of any large-ish corp that tried to use k8s, and they all say "yeah, we tried to write those manifests manually, it was horrible, we now have a home-grown piece of software that generates it" (I mean, don't you like it that your deployment manifest must have metadata.labels, spec.selector.matchLables, and spec.template.metadata.labels all have exactly the same contents or else strange and wonderful things happen?)
Then again, some people make their operators' input language way too flexible so in the end you again have a YAML-serialized AST of an accidentally Turing-complete language... but it's the common ailment that plagues us all: given a language, any language, instead of using it we start building an interpreter for some other ad-hoc language on top of it, and then suffer from the consequences.
As for standard k8s' operators... they're okay, but the moment you need to do anything slightly different from the intended way, you're in a see of hacks and workarounds. For example, you want your pods spread perfectly even between a set of nodes? Haha, why'd you want that, the k8s cluster is supposed to be RAID-like but okay, here's the beta-feature called topologySpreadConstraints. Watch out though: it disregards any inter-pod affinity constraints, and you really want to use it with a nodeSelector/nodeAffinity.
Not the biggest k8s fan out there but I think the verdict is that K8S will stay for a looong time, especially in the enterprise that it's just starting adoption now.
Unless the hype cycle continues + major efforts to improve k8s, I personally don't see it last.
And I'll give you another use-case I'd like to see you replicate with any other existing platform: migrating an entire on-prem application stack of 200+ services to the cloud. It took us 3 weeks total to do for a client of mine, where the hardest part was the migrating state with minimal downtime, and a lot of work went into the IAC part for the GKE setup.
Also, institutions like banks have now jumped on the k8s (well, mostly openshift, but same thing) train, I'm pretty sure it's here to stay.
Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in.
No thanks.
Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in.
> Windows doesn’t have “distributions”, so there’s less need for containers in that space.
I don't follow. What does distributions have to do with need for containers?
A lot (but not everything).
It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc...
In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters.
Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself.
Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable.
[1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is.
[2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc
https://kubernetes.io/docs/concepts/windows/
https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=...
Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios.
DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work.
There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also.
Etc, etc...
https://cloudbase.it/cloudbase-init/
In theory this means they could be managed then by something like KubeVirt instead of treating them like a worker node. https://kubevirt.io/user-guide/virtual_machines/startup_scri...
Excited to see this space continue to evolve for sure.
New Windows VMs are generally built using PowerShell DSC or a similar native tool.
Different distributions run different kernels. A container from a different distribution will probably work, until it doesn’t.
Not so much, no. Linux is the kernel, and all Linux distros, every single one, employs Linux as its kernel, or else it would not be Linux. Stepping back from my pedantry, I believe what you must mean is that different Linux distributions will customize the kernel a bit. That doesn't change what it is. Just because you have automatic windows and I have cranks, and you have leather interior and I have plush doesn't mean our vehicles aren't the exact same year, make and model.
If you run kernel version A in docker, and try and run a container that was designed to use kernel version B you may run into problems.
K8s is not Borg. Not even close. It's a toy that has limited use.
The revolutionary tech is the Linux kernel features that enabled containers, overlay filesystems and docker image format.
The no vendor lock is looks great on paper but you are locked in day one. (Eg on aws you probably use IAM, LB, ASG for K8 Nodes - you can maybe move it to another cloud but the effort is going to be significant). Cloud agnosticism is a lie.
At the K8s level, it seems like the introduction of the Gateway API is probably a good level of abstraction to work towards that will keep things about as flexible as possible without all of the insanity that comes with going beyond that to keep everything 100% vendor neutral.
Even just sticking strictly to the K8s space, there is nothing even close to matching GKE standalone, let alone GKE autopilot that I am aware of. In the world of serverless the gap looks to be even bigger.
I couldn't imagine a situation where I would even consider something else in a greenfield project / company honestly.
At least some of namespaces were contributed by OpenVZ
No one (not enough people?) asked (cared?) for this about the JVM for like 25 years till Linux Cgroups came around and Docker capitalized on it. Its not VM v/s container question that we care about. (I am aware this is a very poor comparison but so is comparing a vm to a container, which is my point)
K8S brings complexity yes, and in a consistent, predictable way. The biggest difference between a k8s based shop vs a DIY VM/Container shop is the former can (in theory) hire some people with intermediate K8S experience, and have them sort out the mess that the people without that experience (or more typically the ones who don't have clear requirements) may have created. With k8s it becomes more predictable to find the boundaries and start pulling things apart in production and make incremental changes. Without k8s, making changes to a production setup is a nightmare that folks who have done that sort of thing don't ever want to repeat.
I've written custom operators and submitted upstream patches to existing-ones, it's not that hard once you know what you're dealing with conceptually.
Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
Kubernetes is not just "a tool", it's a platform. So if you know the k8s fundamentals, being dropped in an unfamiliar environment should not be a problem. It's the same as knowing your way around a Linux system, you might not be familiar with that specific distribution you're thrown into, but you should be able to explore the system and figure stuff out.
> Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
Any engineer who claims to know a bit more than the basics of k8s? It being a software, "devops", SRE, "platform", ... engineer doesn't really matter, I've been labeled all of the above.
I'll never claim k8s is perfect, I have plenty of issues with it, but very few of them are an architectural problem, and I've yet to encounter a platform that attempts to address all the issues k8s does. But you seem to be very critical of it, so what would you propose as a viable alternative?
Edit: I can answer questions on how we do it or maybe work on blog post if you’re seriously asking
How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services) Are you running your workloads in the cloud or on dedicated bare metal machines?
> How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services)
Up to upper hundreds of nodes/tens of thousands of cores (beefy nodes) per cluster in biggest ones (several geo/cloud-distributed clusters). Tbh what’s giving me more grief is number and diversity (different clouds/different setups) of clusters not individual cluster sizes. Original team that built/proved the design was only 3-4 devs but k8s was only part of what we did so it’s more like 1-2 if full time. Number of engineers that deploy there is probably close to 50 now with at least twice as many services (not all actively developed or in-house).
> Are you running your workloads in the cloud or on dedicated bare metal machines?
GCP and AWS VMs. We managed to stay away from anything managed except block and object storage so baremetal is in store for us just wasn’t the time yet. One of my main goals developing our current design was to keep this door open when the time comes and the setup will be largely the same as in cloud (with a few additions like 3rd party CSI for ISCI/NVMeOf storage)
Problems stemmed across multiple dimensions: limits on the numbers of nodes in the cluster (had to split into 2 clusters - think thousands of nodes), crds, weird network setup issues, daemon sets for storage/logs, persistent volumes, rough edges of the tooling operating the cluster (people tell you that with k8s you write less scripts! But in reality depending on your cluster size the scripting moves to the admin side.
This does not include actually upgrading k8s which was a nightmare and having to keep the underlying nodes patched.
> weird network setup issues, daemon sets for storage/logs, persistent volumes, rough edges of the tooling operating the cluster (people tell you that with k8s you write less scripts! But in reality depending on your cluster size the scripting moves to the admin side.
Network with Cilium has been mostly “setup once and forget”. But yeah all the addons will require constant upkeep in a self-managed scenario. As far as configuration management goes I agree it’s a big hole which we had to fill with our own DSL configuration framework (think Helm3 but with Starlark). Now there are products like Pulumi which can offer similar API
A proper golden image pipeline can work well, and you can rightsize your workloads easily.
All of the additional features such as ingress controllers and other complexities make debugging a bit obtuse, especially if you are not extremely proficient.
I could be an old curmudgeon but simplicity is my priority.
I think its vastly more useful to think of a container as an RPM with layered tarballs, a chroot/jail, and some cgroup isolation.
Thinking of them as a virtual machine is the wrong metaphor.
Kubernetes replaces the thing we were doing in 2003 blasting around tarballs of software with scp and some scripts to start and stop things, and is the end result of evolving from that point to a much more dynamic one and with cgroups providing the ability to have software teams not quite clobber each other's workloads while running on the same hardware.
The problems it solves overlaps with many of the problems that VMs were addressing, but it is like looking at two different species that look quite similar but are in completely different animal families and are just similar because of the similar environment they evolved in.
Twitter had Aurora, then elsewhere mesos and mesosphere popped up and it seemed like there were going to be about 100 different frameworks until Kubernetes dropped and basically ate the industry alive. K8s feels pretty stable from where I’m sitting.
It was more like workqueue than borg. But it worked well enough. Eventually we got tired of maintaining our own snowflake scheduler and switched to Kubernetes.
But I'm also old enough to remember pre-borg systems used in HPC. Maui, TORQUE, etc. I wasn't surprised batch scheduling finally got used for service deployment.
It really makes for a constant churn where there's a good change something breaks if you update kubernetes. Either that or you pick a version and stick with it for 2 years while it's supported and then jump to the new one and fix all the breakage.
If i where you i would stop use argumentation's like that.
OpenVMS has still the best Clustering tech...why is "nobody" using it? And btw p9 is very much used (WSL for example).
Linux is for sure not the best kernel, not for server not for desktop and not for mobile...it's good enough and everyone circles around it, it has some bacon for everyone (even if the bacon is sometimes disgusting).
https://en.wikipedia.org/wiki/9P_(protocol)
Same with unix in itself...it's good enough.
Some years ago you could argue that everyone uses mysql...why even bother with that postgresql-thing.
What about something like Hashicorp Nomad and Docker Swarm? The popularity of the former and the maintenance status of the latter aside, I think they achieve much of the same as Kubernetes in ways that are simpler, which is enough for the majority of deployments out there.
For example, most pieces of software that run in containers have a docker-compose.yml file which can then be fed into Docker Compose to launch an environment on a single node, say for local testing and development, or to just explore a piece of software in a throwaway environment. What Docker Swarm does, is take basically the same specification and add the ability to run containers across multiple loads, do networking across them in a reasonably secure way, whilst being able to set up resource limitations etc. as needed, as well as scale the containers across the available nodes and manage storage with volumes, bind mounts or even plugins for something like GlusterFS or just NFS.
Docker Swarm doesn't concern itself with the concept of Pods because you don't always need those - regular containers can be enough without the additional abstraction in the middle. Docker Swarm doesn't concern itself with the concept of a Service, since you can just access containers based on their names through the built in DNS abstraction, especially if you don't need complicated network isolation (which you can also achieve at server level). Docker Swarm doesn't really care about an Ingress abstraction either, since you can just make your own Nginx/Caddy/Apache container and bind it to ports 80 and 443 on all of the nodes where you want to have your own ingress. No PersistentVolume and PersistentVolumeClaim abstractions either, since the aforementioned bind mounts and volumes, or network storage are usually enough. And the resource usage and API are exceedingly simple, you don't even need to worry about service labels or anything like that, since in most cases you'll only care about the service name to access the container through.
If I built my own container orchestrator, I'd strive for that simplicity. Seems like projects like CapRover also recognized that: https://caprover.com/ Same with Dokku: https://dokku.com/
If you're in a company that has never really run advanced A/B tests or doesn't really need to do complex 2-stage DB migrations or blue-green deployments, circuit breaking and other fancy stuff, there's not that much use in going with Kubernetes, unless you really just want to hire for it and also pay someone else to manage it for you.
Personally, with tools like Kompose https://kompose.io/ I'd advise that you start with Docker and Docker Compose at first (locally, or for dev environments) and then branch out to Docker Swarm or Nomad, before eventually migrating over to Kubernetes, if you need to, maybe with something like K3s or K0s clusters at first. Or maybe even Portainer/Rancher, since those make the learning curve of Kubernetes far more tolerable. Or, at the risk of increasing the complexity of your deployments, go with Helm as well because the templates for Deployments and other objects that Helm creates by default are surprisingly useful in avoiding YAML hell.
Of course, some say that Docker Swarm is dead and you shouldn't ever touch it, which is why I mention Nomad (even though HCL is a little bit odd at times), which is also great, with the added quality of supporting non-container deployments. Either way, ideally look for a way to run apps in containers because they feel like the "right" abstraction, whilst finding the best fit for the features that you actually need vs the complexity introduced.
In short: containers are pretty good, running them has lots of options, although some are indeed complicated. Kubernetes is not the simplest way to run them, in my experience.
* Because you are running multiple workloads on the same kernel, you need to protect them from each other, from both correctness and performance perspectives.
* Because the workloads have different scaling characteristics, you need to solve a bin-packing problem to make efficient use of the resources.
* Because of said bin-packing, workloads move around a lot, so you invite a service discovery problem much more intense than classic DNS is meant to solve.
* Because normal software only knows how to use DNS, you invite the need for sidecars and virtual network overlays.
When you have this set of problems then k8s seems appropriate. I work at a company with this set of problems but no Kubernetes, and boy is it a production. But my takeaway from that is less "use Kubernetes" and more "try not to have those problems." Use relatively monolithic architectures, one or a few services on their own stable pools of nodes, with boring old reverse-proxy load balancers and DNS, for as long as you possibly can.
I only know the GCP ecosystem kind of well so I don't fully know to what extent these things exist in AWS and Azure but there I think there is a really nice path you can get on with the serverless route that skips K8s entirely but keeps you very well aligned in case you ever need to "upgrade" or get out of the GCP ecosystem.
I write very stock standard gRPC services and then put them onto Cloud Run (which has a very Heroku like workflow) and stick https://cloud.google.com/api-gateway in front of things and now my API is running on the exact same setup as any other service Google is running in production. Huge amounts of logic get moved out of my code base as a result.
If you are also willing to write your APIs a fairly particular way https://google.aip.dev/ it starts to become trivial to integrate other things like https://cloud.google.com/workflows, https://cloud.google.com/pubsub and https://cloud.google.com/tasks which is traditionally where a lot of the "state" and weirdly complicated logic previously lived in my code. I'm now not really writing any of that.
Now it's all declarative where I just say what I want to happen and I don't have to think about much else beyond that because it too is using that same internal GCP infrastructure to handle all the complicated parts around what to do when things go wrong.
But to me they are all extremely heavily aligned with the K8s path so the lock in certainly doesn't feel as scary.
You have to still ask the question why not just deploy the monolith to the VM and move on until you need to think about anything else?
For me it’s not JUST an investment in the future but in the very real immediate benefits it’s the huge amount of code I don’t have to write and all the insanely advanced stuff I get for free.
To give a quick example of each though. On the code side I don’t think about things like authN, authZ, retry logic, health checking, most security outside of things like input validation, logging, tracing etc. All of that now is just a configuration setting for me.
Then on the advanced features side, just to give one example. If I set a security policy saying this service can do the following actions on another service and I don’t end up using most of those permissions it will automatically notify me in the future to let me know and help me rewrite the security policy to only use the things I need in practice, it can even help me test that new policy to ensure things don’t break… That was previously basically an entire persons job to find and fix those kinds of things, usually manually because the amount of code needed to automate it is painful to think about. Now it’s just another feature I pick up for free.
Yes, and that "everyone" include kernel developers. That's my point. Now, does this mean containers are totally useless, of course not. Especially if you can prevent running attacker code via other means.
Well, not so fast. Ubuntu apport (its coredump collector and crash reporter) had a series of bugs over multiple years which could be used to escape from containers. This is all because coredump filter is still a global thing and not a per namespace one.
I agree with you it's actually more secure than people thought it is. A rule of thumb: just treat it as same as "running multiple applications with different user". Yeah, not a real sandbox. But nowadays also not many low hanging fruits, either.
probably true for the cloud too
Have no fear, these k8s specialists are going to be extremely in demand for getting rid of legacy k8s hell. (Not even joking, I'd pay good money for someone to get rid of the k8s cluster previous consultants have cobbled together.)
You then get the security benefits of a "real" VM while still distributing your software as an OCI (Docker) container, and can operate it like any other k8s cluster.
We are a startup, had zero knowledge of k8s before. Without GKE, we would have lost days and days of work to imitate parts of k8s on VMs.
Scaling up and down, everything as a container (even cron jobs. Otherwise we had to keep a vm running and use some scheduler). GKE has been a massive advantage for us. The complexity bits may be true for someone managing a k8s cluster. Even with just 3-4 services, managing everything as cattle has been amazing.
EKS and AKS are not nearly as well polished and put together, and god help you if you decide to run on-prem.
The problem is of course that you’re quite right that it’s like driving a car, but ultimately unless you have a good support contract with your car manufacturer for a specific use of the car, then you have to take it to another mechanic and they’re going to put you over a barrel.
(Maybe the analogy falls apart here)
Obviously, everyone's use-case is different but none of this has been my experience with VMs:
- Both AWS and GCP allow you to define custom scaling policies for an instance group.
- Deploying new code just means updating a template with your new image and pushing that out. Both GCP and AWS would then cycle the existing machines automatically to use this new image.
- At least in our case, there is no need for Service Discovery; you just need to know where the Load Balancer is.
We have moved some things to Kubernetes and now have to deal with inexplicable occasional failures. That and an everyday massive YAML headache.
YMMV
narrator: it is not. you're not trying very hard if you haven't hit a case where k8s will lead to you banging your heads on any available walls trying to understand a behavior in an edge case (here is one that made walls hate me: https://github.com/kubernetes/kubernetes/issues/96858 and I know of a half dozen developers that learned about this the hard way).
Yes, Docker containers != VMs. That's the point: An old-school VM had all the components needed to run one instance of a service; this might include the core app itself, along with "support processes", like a reverse proxy for TLS termination, and maybe some other services. By design, you cannot run all of those in a single docker container.
The point of the article is to say that Kubernetes pods ~= VMs: A pod has all the components required for a single instance of a service, each one in its own docker container, but with increased access to other containers in the pod (just as processes have increased access to other processes in the same VM). So if you're coming from a VM deployment world, when you see "pod" think "vm", and when you see "sidecar" think "support process".
Sure you can. The notion that “one container, one process” is the biggest misstep of containerization.
> Docker containers have been deliberately designed to contain just one application. One container - one Nginx; one container - one Python web server; one container - one daemon. The lifecycle of a container would be bound to the lifecycle of that application. And running an init process like systemd as a top-level entrypoint was specifically discouraged.
My guess is that the goal of this is to leave "lifecycle" issues at the container boundary. If you put nginx and your golang webapp in the same container, and one of them crashes, Docker / kubernetes you have to have logic inside the container to detect that and restart the correct one. If the goal is to have Docker / Kubernetes is to take care of all that for you, making it one thing per container is the best option.
If the proxy only serves the app you can just shrug and restart the container. If the proxy and the web app are both stateless it shouldn’t matter.
What you don’t want to do is mix stateless processes with statefull services, or have intertwined dependencies.
I spent years abusing docker to run images with systemd in them to test system configuration management tools running as many processes as you like.
Once you hit it hard enough with a wrench you can make it do that just fine.
Makes it very easy and extremely quick to test things in docker containers on your laptop which are meant to eventually wind up on VMs/metal.
Horrible things like entire RHEL distros running systemd in one docker image...
"I watched C-beams glitter in the dark near the Tannhäuser Gate..."
This also makes containers more dev friendly, dev accessible.
I recently learned that one of the heads of SpaceX is heading his own company now. It’ll be interesting to see if his bus number was high enough that the wheels stay on, if the symptoms will be obvious if it wasn’t, or if they will be explained away by other phenomenon. I have had enough successful bosses that credited the wrong actions that it stands to reason that most failures are also blamed on the wrong chain of events. I know my own narratives in 5 Why’s situations are often in stark contrast to other theories.
Is that really a problem? Companies churn people exceptionally fast these days. That inherently involves dealing with the bus problem.
As for 'dealing', look at all the status Google services have lost in the last ten years due to systemic problems with valuing novelty over stability. There are perverse incentives not to maintain things you have delivered, even if you are still at the company.
People forget that mainframe did it first, mainframe still does it best, the only thing that’s changed is that now you can do it on consumer grade electronics for less then 5 million dollars.
A container is an abstract idea that normally comes together through the synergy of [at least] cgroups, namespaces and overlay filesystems.
But i guess i see now what you are saying. That it's a "meta" term for that idea of abstraction.
Hypervisors just seem like particularly impoverished microkernels to me. It reminds me of social programs working hard to avoid the term Socialism because of emotional baggage. Oh this is definitely not a microkernel? Is it a full privilege program that handles a small number of system-wide concerns and delegates the maximum number of other concerns to processes running at a lower privilege level? Yes? Then can we call it a… no? Okay buddy.
Or, "Microkernels done right":
https://www.usenix.org/legacy/event/hotos05/final_papers/ful...
> At the last HotOS, Mendel Rosenblum gave an ‘outrageous’ opinion that the academic obsession with microkernels during the past two decades produced many publications but little impact. He argued that virtual machine monitors (VMMs) had had considerably more practical uptake, despite—or perhaps due to—being principally developed by industry.
> In this paper, we investigate this claim in light of our experiences in developing the Xen [1] virtual machine monitor. We argue that modern VMMs present a practical platform which allows the development and deployment of innovative systems research: in essence, VMMs are microkernels done right.
(Edited to add relevant paragraphs from the abstract.)
It's definitely not the same thing.
Really though, I was just pointing out the tautology of your previous comment. It boils down to "it doesn't run anything that doesn't run on it."
https://stackoverflow.com/questions/4221605/compiling-c-for-...
Indeed I missed that! :)
> You can just compile to JVM bytecode
Perhaps, but it's a hack with no clear advantage, whereas with a container I'm just using kernel features for better isolation and resource management. The application was compiled the way it was meant to be compiled when written.
> the tautology of your previous comment. It boils down to "it doesn't run anything that doesn't run on it."
Perhaps if you take my comment in isolation, but when you put it in context it doesn't.
The thread went:
- Kubernetes reinvented VMs in a good sense (OP)
- I thought we had this kind of covered 20 years ago with J2EE (a rebuttal of OP)
- Doesn't really handle multiple process isolation and IPC as well. (your own comment rebutting the rebuttal :P )
- It also doesn't really handle anything that doesn't run on the JVM. (my comment, adding that it's even farther from the J2EE)
Hopefully this shows how my comment, when taken in context, is not a simple tautology because it adds information.