Learnings from our years of Kubernetes in production
medium.com
medium.com
To avoid chicken-and-egg, your critical services (Drone, Vault, Bind) need to live outside of K8s in something stupid simple, like an ASG or a hot/cold EC2 pair.
I've mostly come to think of K8s as a development tool. It makes it quick and easy for devs to mock up a software architecture and run it anywhere, compared to trying to adopt a single cloud vendor's SaaS tools, and giving devs all the Cloud access needed to control it. Give them access to a semi-locked-down K8s cluster instead and they can build pretty much whatever they need without asking anyone for anything.
For production, it's kind of crap, but usable. It doesn't have any of the operational intelligence you'd want a resilient production system to have, doesn't have real version control, isn't immutable, and makes it very hard to identify and fix problems. A production alternative to K8s should be much more stripped-down, like Fargate, with more useful operational features, and other aspects handled by external projects.
The other capital lesson here? Have backups. The team couldnt restore a bunch of their services effectively, cause they didn't have the manifests. Sure, a managed provider may have less disruptions/avoid some fuckups, but the whole point of Kubernetes is Promise Theory, is Desired State Mamagememt. If you can re-state your asks, put the manifests back, most shit should just work again, easy as that. The team had seemingly no operational system so their whole cluster was a vast special pet. They fucked up. Don't do that.
mere mortals are not privy to all of the internal downstream impacts from that public-facing service outage. it would be like shouting into the void and expecting an answer, and, more, liking it.
no, it is easier to recognize one’s place, pay the tithes, and enjoy one god’s blessings and curses alike. do not stray and attempt to please two, it will only end in misery. (three is right out.)
I'm thinking a lot of that was by design - both Redhat and Google had incentives to get you onto their value-add to get an actual production ready system.
It also created an entire cottage industry, although much of this has faded as everyone moved to purely managed solutions. Because anything else is absolutely insane.
No one ever cares about making tooling in any software project. You’re always using something by a dead-ass random third-party.
Microsoft is probably the company where I am actually using Microsoft-made tools to manage Microsoft-made products. And maybe Adobe back in the day.
And k8s is supposed to make that situation better, but these multi-day outage stories are… common? Why are we adding all this complexity and cost if the result is consumer-PC-tower-in-a-closet-with-no-IAC uptime (or worse)?
Of course, my experience is in no way authoritative, but referencing this type of incident as common is pretty foreign to me and may be mostly relegated to self-managed clusters.
Foreign to me too, but not surprising people report issues as common. there are a lot of footguns in kubernetes that come from a lack of understanding.
You can build a robust kubernetes cluster that hosts an application that’s nearly impossible to bring offline without an act of god, it just takes some know-how and a tiny bit of effort/experience.
Now I'm using Fly.io. They both have their advantages. Folks tend to make kubernetes sound way more difficult than it is. It can be overkill but it can also solve so many challenges out of the box. At least when it's managed. It'll cost you though.
I'm honestly convinced it's half CV-driven development, and half just the fact that it's become the standard workaround for Python dependency hell. Python is still the easiest way to write software, and it's still basically impossible to make an application that works reliably on more than one machine because of how Python dependency management works (or rather doesn't), so you have to use Docker, and apparently Kubernetes is the standard way you deploy Docker containers.
I bet more people actually use docker compose because the buyin is that much smaller.
This is complete bullshit.
Try dotnet then
I'm not advocating for any of those specific solutions and given the choice I would probably use something else, but when I see that my previous CTO wanted kube for single-VM deployments, and a former architect collegue wanted kube for apps that were going to be used by 3 to 5 clients maximum (and in both cases to be run by very small and untrained teams), I think the kool aid has been more than drunk, and I'm now avoiding it like the plague.
For $150 I can fly round trip from New York to San Francisco, on a massively costly and complex giant noisy metal tube with two blades sticking out the sides that are so strong you could put a tank on each one and the blades still wouldn't droop. Why does it have to be so costly and complex, if I could do something simpler, like take a bus? Well, mostly to keep me from dying. But also to carry lots of luggage, keep costs down, and get me there 15x faster.
K8s does provide great value (as a dev tool), but lacks value in production features, and its design is shit. So I wouldn't say complexity and cost are the downside; it's the lacking production value that's the downside.
In production I always want dedicated resources though.
> Automate Software Build and Testing Drone is a self-service Continuous Integration platform for busy development teams.
There's an OSS fork in development (https://woodpecker-ci.org/) but it's far behind in terms of features and stability.
The Root CA certificate, etcd certificate, and API server certificate
expired, which caused the cluster to stop working and prevented our
management of it. The support to resolve this, at that time, in kube-aws
was limited. We brought in an expert, but in the end, we had to rebuild the
entire cluster from scratch.
The control plane can be fully down - as in: you can shut it down - and everything continues to run. I've been in that situation multiple times with large clusters. E.g. one etcd node having disk issues, pretty much turning it into a lame service (even worse than true down). Kubelets got randomly regarded as non-healthy due to update latency. But everything continued to run. Another time API servers were leaking memory, and with that crashing, causing some herding that would crash each server as it comes up. No issues whatsoever, migrate to larger instances.
It's a pretty cool feature of kubernetes.
I am wondering what was done to let this cascade like this. The only thing I could imagine is that someone _wiped_ the etcd state, then brought it up, casing all things to go down.
I wonder how much 'sunk cost' and other psychological factors play into the statement: > We started with Kubernetes a bit too early,
It feels like a "it wasn't that bad". But if you consider the dedicated resources, costs of pain and suffering that may have been avoided with a simpler architecture and infrastructure, I'm forced to wonder if this short comment hides a lot under the covers.
Engineers can have very high pain thresholds sometimes.
Wonder if they're including those engineers in the cost calculation?
So I started setting up k8s clusters on-prem at work in 2019.
It's 3.5 years later and my takeaway is that k8s should only be used after multiple resource planning meetings have established that it is absolutely necessary, or you're scaling up an existing application.
The mistake we made was to first of all take orders from a lead developer who thought that service mesh was the solve all. And secondly to not estimate the load our system would generate, the resources it would require and how to best manage those.
In retrospect we now have 3 clusters (dev, staging, prod) for a service that could be hosted on 2 servers, with a job manager in the cloud.
And IaC is no excuse because you can achieve the same IaC with container hosts and quadlets set to auto-update.
I've found a happy middle ground to be Hashicorp's nomad. It's a single static go binary you can run so you can define some of your setup in a repeatable way as well as provides things like rolling updates, task monitoring, scheduled jobs, etc. And it's not limited to only running containers, but can run executables directly on the host, VMs with qemu, etc.
If I'm running a single server, I usually get a lot of value out of throwing nomad on it.
To me the latter always sounds very unsophisticated.
I don’t think they got replaced. Colloquially they mean the same thing.
Maybe learnings sounds a little more casual and lessons more academic or formal.
Lessons is since the 13th century.
(You know, to a Slack huddle or email thread or something.)
Lessons/learnings
Requests/asks
Solutions/solves
Agreement/alignments
It definitely sounds weird if you don't spend a lot of time in that world. It's like they replace the actual noun forms with an oddly cased verb form, i.e., nominalization.
Oh, one of my most hated:
Thoughts/ideations
Jeez...
Edit: funny how this is down voted but the corporate speak comment not :) we use the word learnings at home. It means what you have learned yourself, as opposed to getting a lesson about it.
Lessons has been used since the 13th century.
It's amazing to think that the regular accepted noun sounds snooty to you.
The least snooty would be "knowledge", which has Anglo-Saxon Germanic roots, and is much older going back to the 12th century.
It might have been spelled "learninges" or "lernynges".
Still, the bottom line is that making nouns of verbs for words for which more commonly used nouns already exist makes a poor impression on many speakers.
I’d say your likelihood of being misread or of giving a poor impression by using the word increases with the distance to the nearest person (other than you) holding an MBA.
"Lesson" by itself might connote a more concrete transmission of knowledge (like a school lesson). Which is a meaningful distinction if the goal of the article is merely to muse about lessons they've learned rather than imply that this is a lesson from the writers to the audience. "Lesson learned" could imply the same thing, but is longer to say ¯\_(ツ)_/¯
I get what the comments here are saying about it sounding corporate, but I think this is a unique situation where this word actually makes sense.
I feel like it's used because "lessons" may imply judgment to some people?
But - Oh boy! The complexity over complexity of moving parts that themselves are a moving target sometimes.
First, there's PKI which you should know all about certificates, signing, expiring, issuing, reissuing or no part of the cluster talks to the other. If you think you can get away with that, the post above had two outages both related to certs.
Next is the etcd - you should know how to configure that. A totally different separate product in its own right, a distributed key store that has whole memory of the system. Like what's where and such.
Then you have whole DNS running. That again is a whole separate product in its own right whose administration you must master or else.
And then comes Networking, the CNI plugin and their internals and if you think you can skip that part, either you have to pay likes of weaveworks (defunct) or Cillium etc or be ready for an incident.
And yet I have not talked about ingress controllers, cloud controllers, their configurations and other issues.
To top that all, you need to manage all that configuration and package it so now you need Helm and Flux - templated (Helm uses Go templates, if not worst out there) and layered (kustomised) YAML upon YAMl, thousands of lines and that's not just all, hold on!
All that configuration language is constantly a moving target from Helm Charts to k8s manifests. Sometimes totally incompatible (like flux to flux2 was almost not so portable) and such so upgrades are going to be so much painful, you just can't imagine even if you're on a managed k8s platform.
I say this from my own experience of setting up k8s self managed from scratch across different clouds. I have my scars.
Thing is, none of these are kubernetes issues. They’re poor practices, these aren’t lessons of running kubernetes they’re poor management of a system.
I traded ELK for Clickhouse & we use Fluentbit to relay logs, mostly created by our homegrown opentelemetry-like lib. We still use Helm, Quay & Drone.
Software architecture is mostly stateless replicas of ~12x mini services with a primary monolith. DBs etc sit off cluster. Full cluster rebuild and switchover takes about 60min-90min, we do it about 1-2x a year and have 3 developers in a team of 5 that can do it (thanks to good documentation, automation and keeping our use simple).
We have a single cloud dev environment, local dev is just running the parts of the system you need to affect.
Some tradeoffs and yes burned time to get there, but it's great.
In the blog, there are multiple days of downtime, a complete cluster rebuild, a description of how individual experts have to be crowned as the technology is too complex to jump in and out of in any real production environment, handling versioning of helm and k8s, a description of managing the underlying scripts to rebuild for disaster (I'm assuming there's a data persistence/backup step here that goes unmentioned!), and on, and on and on.
When you're already using cloud primitives, why not use your existing expertise there, their serverless offerings, and learn the IaC tooling of choice for that provider?
Yes it will be more expensive on your cloud bell. But when you measure the TCO, is it really?
> While being vendor-agnostic is a great idea, for us, it came with a high opportunity cost. After a while, we decided to go all-in on AKS-related Azure products, like the container registry, security scanning, auth, etc. For us, this resulted in an improved developer experience, simplified security ( centralized access management with Azure Entra Id), and more, which led to faster time-to-market and reduced costs (volume benefits).
I've seen both work in completely separate domains (e.g. inference on real time video streams vs. model building) -- but obviously ymmv, tech is a big domain and pretending I understand exactly what you're doing would be silly. Sometimes there is a real answer to the why!
I’m new to this stuff, so it’s hard to dig through all of the possible different solutions.
I looked into Ray a bit but it seemed a little too complicated vs. just running a CUDA accelerated docker container. Most of the streamlined solutions in this space are not made for full stack web developers deploying a service that happens to need a GPU. They’re for ML devs who are trying to own the production side of their part of the product.
And in smaller setups you would have a shared cluster or fully managed like gke etc.
However, I also still use Kubernetes for my personal projects, because I really appreciate the level of abstraction it supplies. Everyone always points out that you can do all the things k8s does in other ways, but what I like about it defines a common way to do everything. I don't care that there are 50 ways to do it, I just like having one way.
What this allows is for tools to seamlessly work together. It is trivial to have all sorts of cool functionality with minimal configuration.
Yes. You have to understand that a lot of people without the benefit of experience will often base their technology choices on blog posts. K8S has a lot of mindshare and blog attention, so it gets seen as the only way to run a container in a production environment, while all the important aspects of it are ignored.
Yes, I am experienced and smart enough to know the statements that don't apply to me. My frustration is that I want to discuss the best tools and techniques the industry is exploring, but every time I start to have those conversations, someone comments that I don't need it.
Places like hacker news, or reddit, or twitter, are all full of random people, many of whom are just beginning their journey. Recommending multi node orchestration when they'd struggle to get nginx running on it's own, would be inappropriate. They don't need k8s. There's a significant danger of cargo culting here.
which are?
I am seriously asking. I use docker-compose of some of the things I do but it never occured to me during my 20 years in systems engineering that k8s offers any kind of great abstraction. For small systems it is easy to use docker (for example running a database for testing). For larger projects there are so many aternatives to k8s that are better, including the major cloud vendor offerings that I have really a hard time justifying even to consider k8s. After years of carnage that they left, seeing failures after failures, even customers reaching out to me in panic to help them because there are timeouts or other issues that nobody can resolve after selling them the idea that k8s has "great level of abstraction" and putting it to production.
> I don't care that there are 50 ways to do it, I just like having one way.
Seeing everything as a nail...
> which are?
When I am creating a new service/application, I just need to define in my resource what I need... listening ports, persistent storage, CPU, memory, ingress, etc... then I am free to change how those are provided without having to change the app. If a new, better, storage provider comes along, I can switch it out without changing anything on my app.
At my work, we have on premise clusters as well as cloud clusters, and I can move my workloads between them seamlessly. In the cloud, we use EBS backed volumes, but my app doesn't need to care. On the on-prem clusters, we use longhorn, but again my app doesn't care. In AWS, we use the ELB as our ingress, but my app doesn't care... on prem, I use metallb, but my app doesn't care.
I just specify that I need a cert and a URL, and each cluster is set up to update DNS and get me a cert. I don't have to worry about DNS or certs expiring. When I deploy my app to a different cluster, that all gets updated automatically.
I also get monitoring for free. Prometheus knows how to discover my services and gather metrics, no matter where I deploy. For log processing, when a new tool comes out, I can plug it in with a few lines of configuration.
The kubernetes resource model provides a standard way to define my stuff. Other services know how to read that resource model and interact with it. If I need something different, I can create my own CRD and controller.
I am able to run a database using a cluster controller with my on prem cluster without having to manage individual nodes. Anyone who has run a database cluster manually knows hardware maintenance or failure is a whole thing... with controllers and k8s nodes, I just need to use node drain and my controller will know how to move the cluster members to different nodes. I can update and upgrade the hardware without having to do anything special. Hardware patching is way easier.
The k8s model forces you to specify how your service should handle node failure, and nodes coming in or out are built into the model from the beginning. It forces you to think about horizontal scaling, failover, and maintenance from the beginning, and gives a standard way for it to work. When you do a node drain, every single app deployed to the cluster knows what to do, and the maintainer doesn't have to think about it.
>> I don't care that there are 50 ways to do it, I just like having one way.
> Seeing everything as a nail...
I don't think that is a fair comparison, because you can create CRDs if your model doesn't fit any existing resource. However, even when you create a CRD, it is still a standard resource that hooks into all of the k8s lifecycle management, and you become part of that ecosystem.
These exists without k8s. I do not need a compex abstraction hiding the ways I need to talk to persistent storage. If fact, I believe it is impossible to create such abstraction without very serious compromises.
> In AWS, we use the ELB as our ingress, but my app doesn't care
Your app does not care without k8s. Running python -m http.server does not even know what ELB is. I get it though. You are using k8s as IaC.
That's pretty interesting take considering EBS is itself a block device abstraction over network attached storage and pretty complex at that too with a huge price premium
That was exactly my earlier point... of course you can do everything in k8s in other ways, but in the end you have to pick ONE way your company/team is going to do it... why not pick a well defined way, that new hires can already know, that has a ton of tooling available, and works together cohesively?
Yes, I can build each part myself, but why?
> Your app does not care without k8s. Running python -m http.server does not even know what ELB is. I get it though. You are using k8s as IaC.
Sure, but I still need a way to deploy my app, and to move it to a different location when I do hardware maintenance, and a way to get a DNS address that routes to my app.
At my shop, using k8s, I can deploy a brand new service, with a cert, a url, and a place to run it, in a few minutes. I don't have to talk to anyone, I don't have to use any other tools or have to click on any buttons, i just helm install or kubectl apply and my service is running. I don't have to ask the datacenter ops people to find me a server, or get budget for a new AWS instance. I can deploy to an existing cluster and use a small bit of the infrastructure. I don't have to scale my individual service, I can scale the whole cluster for all services.
It is just so much easier to be a developer in this world.
As someone who is very pro cloud -- one of my worst experiences working at a cloud provider was a push from on high to sell our customers on a 'cloud modernization initative' that centered on managed kubernetes. At the time, most of my customers were struggling with creating a stateless app, much less horizontal scaling and managing an enterprise-grade compute abstraction layer.
I think K8S is a great tool with a dedicated team and a platform built around it to meet the way that your company ships infrastructure. But what I've just mentioned only makes sense fiscally in the high X00's count or more of engineers.
Of course. You're going with what you know for the foundation, then building the thing you're interested in (personal projects) on top of that.
It's a good way of using your time efficiently. Nothing at all wrong with that.
I think size of team matters as the impact of k8s ownership as a fraction of your development velocity changes immensely as you're able to afford a platform team who can build tooling to remove the cognitive load of deploying to and managing k8s. At an ~400 engineer company I worked at, k8s bugs that actually impacted our team were in the single digits over a year, but a large part of that was the platform team that managed the ecosystem around k8s deployments.
ECS and Container Apps provides all the benefits of k8s without the cons. What we want is a to be able to execute container (Docker) images with autoscaling and control which group of instances can talk to each other. What we do not want to do:
- learn all of the error modes of k8s
- learn all the network modes of k8s
- learn the tooling of k8s (and the pitfalls)
- learn how to embed yaml into yaml the right way (I have seen some of the tools are doing this)
- do upgrades of k8s and figuring out what has changed the way that is backward incompatible
- learn how to manage certificates for k8s the right way
- learn how to debug DNS issues in a distributed system (https://github.com/kubernetes/kubernetes/issues/110550 and many more)
I could go on and on but many people and companies figured out the hard way that k8s complexity is not justified.
Maybe I'm biased and just have the wrong kind of projects, but so far everything I encountered could be built with a simple tech stack on virtual or native hardware. A reverse proxy/webserver, some frontend library/framework, a backend, database, maybe some queues/logs/caching solutions on any server Linux distribution. Maintenance is minimal, dirt cheap, no vendor lock-in and easy to teach. Is everyone building the next Amazon/Netflix/Goole and needs to scale to infinity? I feel there is such a huge amount of software and companies that will never require or benefit from Kubernetes.
I think the cost of self-managed Linux VMs and monoliths is smaller than the cloud vendors made it seem.
Containers are nice when you have to deal with a language like Python and it's packaging ecosystem, but when Go/Rust/.Net/etc binaries are placed in containers as well... I think sight of what we're trying to solve in real life has been kind of lost.
I'm in the ML space and every small company I try to avoid EKS. Then I hate my life. Sagemaker, for example, is a giant abstracted away mess with random holes (ie: these types of jobs don't work on this GPU type, etc.) compared to just running things on EKS. The same goes to trying to deploy a more complex third party application. I could just deploy their Helm chart or I could spend a lot of time deploying it somehow in our environment.
I am super pro Ray for handling these types of workloads now. Huge shout out to anyone here working on maintaining that project.
Right now I'm working a lot with an oversized maven configuration that nobody understands ; I'm paid only to dig into it and maybe refactor some parts. It's made way too complicated for the task and does a lot of non-standard stuff to work around problems it created itself. But when I arrived people were blaming jenkins and wanted to move to gitlab because jenkins was becoming too complicated to work around maven (also !). Next thing you know somebody could try kubernetes or moving to the cloud or switching from RedHat to NixOS or whatever, and the problem would still be maven.
Admitting that you need at least two full-time engineers working on Kubernetes I wonder how that kind of investment pay’s itself back, especially because of all the added complexity.
I desperately would like to rebuild their environment on regular VMs, maybe not even containerized and understand what the infrastructure cost would have been. And what the maintenance burden would have been as compared to kubernetes.
Maybe it’s not about pure infrastructure cost but about the development-to-production pipeline. But still.
These is just so much context that seems relevant to understand if an investment in kubernetes is warranted or not.
Why not just have auto-scaling servers with a CI/CD pipeline.
Seems so much easier and more convenient.
But I guess developers are just always drawn to complexity.
It's in their nature that's why they became developers in the first place.
Not sure why you think that?
Developers who like simplicity are a thing, and they seem to really think about how the parts of their systems interact.
What I do notice - and I'm not saying this is true for you (I don't know) - is that people get excited about technology like kubernetes or something equivalent, but it creates an additional burden that is totally not proportional to the benefits of said technology.
Stupid load-balancing proxy servers with a bunch of hosts behind them is extremely uninteresting and boring. But it's dead simple, so reliable and easy to scale horizontal or vertical as well.
And most of all: how much do you save by dynamically upscaling and downscaling as opposed to just keeping a static environment.
If you want to share that kind of perspective, I'd rather see a public post about it that others may benefit from.
I would even say that there is less work with Kubernetes, but maybe that's my preference. I don't even think that you need two full time engineers working on it constantly, or more to say, if you're working on it constantly you have bigger problems but not with k8s. Sure, you need people to own it, but the work is periodical (mostly for cluster upgrades which are too frequent IMO) and in non-toxic companies there is always good work to be done.
Things on my team's todo list (aka: correct the sins that occurred before therealfiona was hired):
- Change manifest files over to Helm. (Managing thousands of lines of yaml sucks, don't do it, use Helm or similar that we have not discovered yet.) - Setup Renovate to help keep Helm chart versions up to date. - Continue improving our process because there was none as of 2 years ago.
We upgrade everything once a quarter. Usually takes about four hours per cluster. Occasionally we run into something that's deprecated and we lose another day, but not more than once a year.
One technique is to never upgrade clusters. Instead, create a new cluster, apply manifests, then point your DNS or load balancers to the new one.
That technique won't work with every kind of architecture, but it works with those that are designed with the "immutable infrastructure" approach in mind.
There's a good comment in this thread about not having your essential services like vault inside of kubernetes.
If Kubernetes is only for stateless services, well, that's much less useful for the org to invest in.
In that case you don't want to migrate application/user data so you are kind of forced to keep DBs and filesystems outside.
- create a dev cluster
- create a test cluster
- manage dev & test clusters with gitops tool
- copy payloads and objects from your prod cluster in the gitops tool
- compare test to prod. try new things out in dev. tear down dev cluster at will
- create a new prod cluster based off of test cluster
- migrate things over slowly
Kubernetes is a perfectly good place to keep your desired state. I think it would be in most people’s best interests to learn to maintain, failover, and recover Kubernetes clusters, so they can trust the API, rather than trying to use tools to orchestrate their orchestration
> learn to maintain, failover, and recover Kubernetes clusters
Kubernetes administration is a completely different topic to gitops, the two compliment each other - they don't replace each other.
Assuming you have some sort of build pipeline that also pushes to your cluster, Flux does the same thing whilst ensuring whatever was pushed never diverges.
a cluster manager isn’t cheap but it sounds like you are getting buried. If you’re on 1.23 or up though, you at least have a year now to fix it.
The control plane is ha and you can upgrade one after the other independent of your workers.
With workers you can do that too.
That feels much less like a snowflake and more like snow.
Even we run into that issue 5 years ago.
But k8s is still very young and the problem is solved for 5-6 years
This doesn’t always work and the failure mode of committing to this can be doubling your hosting bill if it won’t run locally and densely packed small instances can’t handle your app.
When it works it's great.
I _think_ this space just hasn't matured enough to have settled on standard solutions. Between dev, CI, and staging/prod, it feels like you're defining variants of the same objects and topologies over and over. Some things try to take on dev+CI or CI+deploy (dev+deploy anyone?), but I haven't seen an answer for the whole thing that "feels right".
Yeah we do. We use k3s, and use kustomize to scale down or disable services from prod. minio replaces S3, postgres runs nicely in a container, environment variables for service discovery. Add in a liberal sprinkling of NodePort services to allow connections from web browsers or whatever.
Been using it for about three years now. Works great. Upgrading is easy, rebuilding is easy.
We even use it to run our one-box testing environment on an EC2 instance, rather than spinning up an actual EKS cluster. Also works great.
I love that we use the exact same tools to manage all environments -- yaml+kustomize for manifests, kubectl+k9s for ops -- so we get really used to using them.
I've run into this and learned my lesson/gained my battle scars, but it just seems like unnecessary pain. Would it have been so bad for k8s to use something simple for securing communications other than the full TLS stack, right from the beginning?
It's so cumbersome and so many people run into this footgun that also happens to be proper security practice.
A symmetric key setup is simple and if it was available as a fallback all this pain could be avoided. It's not as secure, and you have to be careful with nonces and things but I'll take some somewhat distant insecurity (if someone is already inside your network and reading your asymmetric secrets you have other problems) for the better ergonomics and lower likelihood of blowing off my own foot.
The difficulty is securing comms between components (assuming they can reach each other, just making sure that the payloads are secret) and making sure you don't leak secrets unintentionally (forgetting nonces) and all the other hard crypto things.
But, it's not impossible to make a reasonable to use fallback system that does this, just no one does because of fear of being mocked for not just accepting the pain and bad ergonomics of TLS.
Other systems do work, but they have the footguns mentioned in the article that everyone seems to hit.
In a nutshell, I’m looking for a general way to provide compute and storage to future, modern, applications and engineers, while at the same time training them to manage this themselves. It’s a medium-long term thing. The scale is already there - one of our goals is to replace an application with millions of existing users.
Importantly, the company wants us to be self sufficient. So a RedHat contract to manage an OpenShift cluster won’t fly (although maybe openshift itself will?)
For the specific goals that we have, the broad features of Kubernetes fit the bill - in terms of our ability to launch a set of containers or features into a cluster, run CICD, run tests, provide storage, host long- and short lived applications, etc. But I’m worried about the complexity and durability of such a complex system in our environment - in the medium term, they need to be able to do this without me, that’s the whole point. This article hasn’t helped me feel better about k8s!
I personally avoided using k8s until the managed flavours came about, and I’m really concerned about the complexity of deploying this, but I think some kind of cluster management system is critical; I don’t want us to go back to manually installing software on individual machines (using either packaging or just plain docker). I want there to be a bunch of resources that we can consume or grow as we become more proficient.
I’ve previously used Nomad in production, which was much simpler than K8s, and I was wondering if this or something else might be a better choice? How hard is k8s to set up today? What is the risk of the kind of failures these guys hit, today?
Are there any other environments where I can manage a set of applications on a cluster of say 10 compute VMs? Any other suggestions?
Without knowing a lot about their systems, I suspect something like Oxide might be the best bet for us - but I doubt we have the budget for a machine like that. But any other thoughts or ideas would be welcome.
One of the main drivers of the project is to reduce our reliance on software vendors and move to open source solutions, so a hybrid/on-prem cloud vendor is probably not on the cards for us either.
> I doubt we have the budget for a machine like that.
Before even thinking about budget,
> for a significant national enterprise in a developing country.
I suspect we just aren't ready to sell in your country, whatever it is, for very normal "gotta get the product certified through safety regulations" kinds of reasons. We will get there eventually.
buuuuut also,
> Are there any other environments where I can manage a set of applications on a cluster of say 10 compute VMs? Any other suggestions?
Oxide would give you those VMs, but if you want orchestration with them, you'd be running kubes or whatever else, yourself, on top of it. So I don't think our current product would give you exactly what you want anyway, or at least, you'd be in the same spot you are now with regards to the orchestration layer.
Totally get it re certification etc. There is probably some kind of bilateral standards arrangement with one of the neighbouring countries but I agree with your general thrust - we’re a long way from the point where that’s actually a consideration.
EDIT: seems the limit is no longer 16 per cluster https://pve.proxmox.com/wiki/Cluster_Manager
I’ll check out cloudstack. Thanks!
Again this is really more of a vmms setup so you need to orchestrate on top. (So perhaps your nomad cluster can sit on this (assuming there is some customer/org infrastructure splitting)
Most companies don't reach there.
In general however, post Nomad licensing - there's a gap for an open source cluster orchestrator that is not k8s and isn't Nomad.
I also thought it was EOL, did something change?
Probably true for many early stage projects.
That's crazy, I've personally recovered 1.11-ish kops clusters from this exact fault and it's not that hard when you really understand how it works. Sounds like a case of bad "expert" advice.
> This is very context-specific, but depending on the node type, AKS reserves about ~10-30% of the available memory (for internal AKS services)
25% of the first 4GB of memory, 20% of the next 4GB of memory (up to 8GB), 10% of the next 8GB of memory (up to 16GB), 6% of the next 112GB of memory (up to 128GB), 2% of any memory above 128GB
"AKS reserves an additional 2GB for system process in Windows nodes that are not part of the calculated memory."
https://learn.microsoft.com/en-us/azure/aks/concepts-cluster...
I believe certs were optional back then so no need to rotate!
Please please spare the world more corporate-speak h*rsesh*t. Waves fist at cloud.
Don't use selfmanaged k8s.
From my perspective, the root issue is that people are using the wrong tools for their projects, namely the wrong programming language.
The problem started when people began abusing scripting languages for something else than scripting.
Python was meant as a teaching language for kids. JavaScript was meant to for some gimmicks on Websites. .net was meant for UI applications. PHP stood for "Personal Home Page Tools".
But somehow people started using these tools for something completely different than what they were meant for.
Due to people abusing those languages to write server backends, it suddenly became a problem of those languages creating code that's 1000 to 75.000 times slower than native code.
That then brought the need for clusters, load-balancers, etc. And that the need for tools to manage those. Then they needed tools to manage those tools. And tools to manage the people who manage those tools. And every year I look at the industry, another layer of complexity is needed.
So, the author writes: "Our platform back then was 50% .Net and 50% Python" - and here is the ACTUALY learnings he should have taken away:
"Eight years ago, after our 'developers' had drafted our products in scripting languages, we hired a senior C/C++/Pascal/Rust programmer. That developer re-wrote our drafts in a clean way. He used a profiler and checked for memory leaks, and did some optimizations. Afterwards we bought three servers in two different data centers with two upstream providers each. Since then we had zero downtime, our servers are only consuming 800 Watts in total, our operating costs are minimal, and we are contributing towards having a greener planet. Looking at our company growth rate, those six servers will be good for another 8 years."
Think that's hyperbole? No, it's not. If your interpreted scripting language is 1000x slower than a compiled one, you'll simply need 1000x the resources. You are burning our planet and your money just because you weren't able to accept the fact that it's OK to use a scripting language for quick hacks and... scripting, but that it's the wrong tool for high workloads.
So, for your next project, please consult this checklist:
[ ] Is the project about teaching kids how to program? [ ] Is the project about adding a blinking button to your website? [ ] Is the project about running a desktop application on a Windows PC? [ ] Is the project about doing your personal home page?
In case you have not checked any of the above boxes, you might want to consider having your code re-written in native code, avoiding 90% of your management layers and dependency hell.
And as a bonus, with the CO2 you have saved the planet you are able to buy another two SUVs! ;)
vs
"Eight years ago, our engineers decided to settle on Rust, and six years ago we went out of business, because it was much harder to write web services quickly that met what customers actually wanted"
Good thing our control planes are now managed. I learnt so much about kubeadm and the inner workings of k8s but I'm not sure I wanna go over it again
Spare us all the corporate-speak horseshit. Waves fist at cloud.