Kubernetes Exposed: One YAML Away from Disaster
blog.aquasec.com
blog.aquasec.com
I brought the problem up with the higher-ups and they were pretty lethargic about it. Moral of the story I guess is if you're gonna choose to operate infrastructure like Kubernetes make sure you have experienced people trained to do so and don't expect your devs to wear another hat.
There’s no way an app developer is going to be able to do their work while also knowing the ins and outs of cluster mgmt.
Ya, and everywhere I've worked, that team is called "devops". The application developers are a different team. Maybe that's not the norm, but I thought it was.
But then that team is only doing operations and not development. Unless they are committers to k8s.
I started back in 2016 building k8s clusters from scratch via terraform and bash scripting. It was ugly awful but worked thru Covid with zero downtime when we switched to hosted solutions from AWS and Google cloud.
What a lot of people miss about larger k8s deployment is that you can create an internal platform team, like a micro-heroku within the company. The company would have to hire people who can create _products_ and also knowing how to work with infra in order to create a platform. It would be this rare combination of product, dev, ops, and focusing on supporting the rest of the engineering team and giving up on the glory of launching something for end users.
In the early days, computers were huge things that were maintained by a team of specialists (and who actually soldered things together). Getting anything changed involved a series of paper forms and waiting for a specialist to do it for you.
Then came the microcomputer revolution and computers became small things that sat on desks and were maintained by the office geek. Changing anything meant talking to him (it was always him) and it would be done by lunchtime, and you owed him a coffee.
Then companies learned that this meant that no two offices had the same configuration, application suite, or even operating system. IT was centralised, and every office issued with the same stuff. Getting anything changed meant filling in forms and waiting for System Administrators to get around to it.
Then Agile and the Cloud happened and while the rest of the company had to carry on filling in forms, the IT Department turned into Engineering and DevOps was born. Every developer learned how to deploy their stuff to the cloud by themselves.
Then organisations worked out that that meant that applications were deployed badly, using a wide variety of tools and technologies, and essential security practices got skipped because no-one was responsible for them. So DevOps was turned back into SysAdmin and the forms reinvented as JIRA tickets.
And so we go on. There are micro-cycles within these ones, like the one about terminals (are you typing on the actual computer, or on a terminal that talks to the actual computer which is somewhere else? The answer depends on which decade you're in).
The ideal solution is somewhere in between (enough central control for it not to get into a mess, but enough decentralisation to prevent bureaucracy and allow for agility in the actual business). But that is a very hard balance to find and keep.
I feel similarly about fullstack. You’re a front end dev that has written a controller and used an ORM before. Often not too inspiring. Backend devs are still needed, whereas full stack devs form excellent glue.
That’s what DevOps is… or, was. I remember when it was new; that was the whole thing: dev and ops should work together. A break from the norm of the time that dev throws it over the wall to ops and never thinks about it again. What a tragedy that it’s been cargo-culted into its current meaning of “devs do ops” (poorly).
Anyway, cheers. In short, I agree with your take.
In my experience, devops folks usually come from the dev side, not the ops side, and their experience and curiosity reflects that. So we let them focus on shipping line-of-business code while making sure what they ship runs in a competently managed environment, as well as handling data centers and all the more traditional infra stuff.
Also, I don't have many resources at my disposal, k8s requires a separate storage cluster. All-together I would've required 9 servers, 3 control pane, 3 app or pods, 3 storage. Way too expensive. And then think further, all this has to be updated sometime. I don't have the brain and time resources to do that.
So, if I want k8s, I need to pay for expensive cloud. Or, I can remain with my 1 server with backups and build systems on demand. Yes, I hear the booksmarties scream "but but failsafety". 1 incubator server, specialized systems when something takes off.
Some companies can afford to pay for expensive cloud, more power to them. I can't. I'm busy solving problems at hand, not dealing with additional problems of a system I don't fully understand.
Yes, k8s is a nice comfortable way, if you can afford it. But it's expensive and wasteful of resources. And we do have a climate crisis.
Everyone assumes you have to run multi node.
You can run each of those roles on the same machine, achieving high availability from 3 total. For learning purposes you can run this all in VMs on a single machine.
There are plenty of good reasons not to try kubernetes when it doesn’t fit your use case, but the overhead it imposes is mostly in time and expertise. Claiming it’s more costly in compute resources, and especially contributing climate crisis, is reaching.
Why? Because Kubernetes has some great ideas on self-healing and resiliency.
But if you are small, and can work with a single instance for availability, then probably should stick with systemd. You could also use podman to get a small bit of the kubernetes ideas for working with containers.
The next level of scale wouldn’t be the full k8s but more like k0s, or k3s.
I'm working on a product that uses something like this internally, basically, we provide multiple preview-environments in the same big server, these are managed by systemd.
This is the first time I get to think that this may be a project by itself, ping me if you have any ideas or want to get into a discussion on this topic.
Kubernetes does not require storage cluster. If you need it, you can run it. But it's not required.
Please think about your arguments when comparing kubernetes with any other solution. If you don't need storage cluster with any other solution, then you don't need it with kubernetes.
Kubernetes at its minimum is 3GB RAM and 0.5 CPU for one control plane. That's all you need. Everything extra is not necessary.
People often try to build some kind of super-resilient cluster with kubernetes, which is suitable for huge company with hundreds of servers, while comparing it with one host docker compose. If your task is simple, don't make it more complex than necessary. Kubernetes can be simple or it can be complex, it depends on you.
What you do need for kubernetes is knowledge and experience. I spent several months tinkering with cluster before I dared to deploy something valuable there. Although I used kubeadm, with cloud offering few things are simpler.
That is only half of the truth. If your application needs a database, and which application doesn't, you have to store its data somewhere. So you either need a storage cluster, or you open the can of warms called local volumes and (anti-)affinity rules.
For smaller application deployments, hosting a separate kubernetes cluster is extremely expensive in terms of time spend to debug infrastructure issues. k8s introduces orders of magnitudes of additional complexity to networking, dns, storage, routing, logging, etc. All in the name of making these exact same things easier for the developer. This can be a good separation of work, if you are large enough to afford a dedicated k8s maintenance team. But often it is not.
Regarding storage, try to build stateless apps that only need temporary storage that don’t need to outlive a request (like uploads). For long lived unchanging assets, you can try including in the container (ie. defaults.json, countries.json, samples.json, etc). For assets that need to be fresh, try to have the app generate at startup (eg. Email vírus definitions) or via init container. I think this will cover most use cases.
This won’t work with apps that depend on shared storage like Wordpress, but Wordpress is horrible and I can’t support you running it :-)
So if you think that you need storage cluster either way, go ahead and deploy it. If you think that you're OK with single unreplicated database, just use local volume, it works. At least I've yet encounter any worm with this thing. For me it just worked.
Another option is to use local volumes and replicated database setup. So you can run, say, two database instances in a replicated mode on two servers, but storage will be simple local volume. It provides some reliability and availability benefits without complex storage cluster solutions. I used cloudnative pg operator for this and it worked for me. I guess one can configure ordinary postgres replication without any operator, I just don't have experience with that.
My opinion is that even single-node kubernetes cluster is worth it. That's because I have some experience and won't need time to learn basics. If someone does not have that experience, then I agree. Deploying kubernetes requires learning curve and that time might be spent for other things.
At least it's hard for me to imagine any kubernetes complication that won't arise with, say, docker. At the same time, kubernetes provides lots of polished software line ingress-nginx, cert-manager, cloudnative-pg which could save some time. And, most importantly, using kubernetes from the start, provides very easy option for moving into cloud, for scaling, for zero-downtime deployments. Those things are valuable by almost everyone.
Your control plane and worker node can be the same machine.
Rancher + Rancher Kubernetes Engine made it a lot easier for us to manage as maintaining that is simply "run this docker container" and it deploys all the k8s gubbins for you. And k3s is a lot more lightweight on resources than k8s. Stuff like certificate rotation and kubernetes version upgrades are handled for us, it's reduced to clicking buttons on UI.
Storage can be local, it's just not / wasn't well explained in the documentation. PersistentVolume[Claim]s can use directories on the host, those PVs/PVCs can be shared between many pods and you can mount subdirectories into each pod using `subPath`.
You can get it all running on a laptop using a couple hundred MB of memory, all in all. So, not going to be ideal for running on a Raspberry Pi, but on par with a lightweight linux desktop environment in terms of resource usage.
I hate the complexity sometimes, but then I still end up having the same learning curves with things like SystemD when I use them anyway - every "run a bunch of stuff on my machine and restart it if it breaks" system seems to be rife with edge cases and behaviours which you have to dig deep into the documentation to see - and we don't _want_ to RTFM if we don't expressly _have_ to :-)
^ Although, okay, SystemD doesn't require you to have random HTTPS authenticated endpoints to talk to it and TLS certificates which expire every year and consists of _one_ init process rather than like 7 interdependent docker containers which can restart at any time and rely on container infrastructure to be running and argh. Okay yeah the flashbacks are happening and I agree with everything you've said again.
a foss alternative is kubeztl - it takes kubectl off the networks - no public IP at all - but w/o users needing agents: https://github.com/openziti-test-kitchen/kubeztl/tree/main
disclosure: i am a maintainer and the software overlay in the middle (helps enforce outbound-only, pre-authorized connects only) needs to be managed (self-hosted foss or hosted saas), so there are still trade-offs.
I instead started writing my own product for pull request assignments and direct Slack notifications for those (my personal frustration).
I needed to get away from it all and decompress. It's been refreshing writing code.
So what is growth? The number of developers you have doing stuff. You hire more and more, and every few months you parade some Patagonia vests through your offices to show them. "Observe, more devs! Growth!"
The problems is, the devs all have to do something.
I stopped feeling productive years ago. It's all just never-ending grind, dealing with complexity that has no f--ing business being there. And the younger engineers have no idea. They think this is normal.
This is mostly what we have done. We want to make money. There are barely 10 of us. We don't have time for make believe work that doesn't bring in extra revenue. We used to play in traffic with conversations like "micro services" and "one repo per service" but our investor killed that mode of business and gave us one more chance to do it right.
I recall the original reason for moving away from one big database being something approximating "it won't scale". How well are those various k8s contraptions scaling relative to the "legacy" verticals they purport to replace? How many of us would go back in an instant if given the opportunity?
I think the biggest fear for most on HN is going through ego death over this. The fact that even your "hard" problem is trivially solved with boring tools that were invented before you were born.
To test this, if you hit the K8s API server unauthenticated, do you get a 401 error or a 403 error? If the latter, you're at risk.
Makes it easy to find out what clusters are running what versions of k8s which is interesting, but not a major security issue.
Whilst it usually was just bound to localhost, I did encounter a distribution that, by default, bound it to the container network, meaning that anyone with access to one pod, got to be cluster admin!
Generally speaking though I don't think any distributions do that now.
Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default.
I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.
There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant.
It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.
Also, it doesn’t help that the kubernetes documentation is awful.
I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.
You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production.
With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run.
Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox.
K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".
Carefully manage complexity, and write a good manual.
OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does).
I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree.
Sources:
https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li...
Edit: Using open hub for both says OpenBSD is 75M lines:
My mind immediately went to Eddie Izzard’s “do you have a flag” bit.
> It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.
Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.
Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production.
But thanks for replying, anyway.
Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.
Nothing in the middle.
There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website.
The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.
Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).
The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.
I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking.
One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it.
There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it.
https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about.
It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
Kubeadm is a pretty barebones option, which I think needs work to scale (you're on the hook to add all the bells and whistles to create a full k8s environment)
There's validating admission policy coming along (https://kubernetes.io/docs/reference/access-authn-authz/vali...) beta in 1.28, but you still need the policies to apply.
Really for a more "batteries included" kind of cluster, you'd need one of the larger distributions that tries to do more out of the box. Obviously OpenShift is the ultimate example of that (for on-prem) but it massively increases the complexity of the environment.
Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.
Of course not, unless you configure it this way.
> RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security.
What? How comes no one uses service accounts? They're used extensively for many things.
K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.
RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s)
Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common
Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now.
I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)
Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings.
The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS.
Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege.
Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev.
You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.
The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point.
I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references.
for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.
I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box.
I'll stay with swarm for now
I think they were saying that is akin to the state of kubernetes, where everything is too new to really judge it today. Let me make clear my bias before I get to my opinion: I'm a big fan of what kubernetes has accomplished with managing the deployment of applications. But I think comparing current day kubernetes to a restaurant's grand opening is naïve. Kubernetes is amazing at what it does, and for how highly flexible it is. But it is highly flexible because of how advanced it has become, which in turn requires more knowledgeable administrators. You can take off with Docker Swarm because... it's very simple. It doesn't aim to be flexible, it just aims to be the docker-compose of distributed systems. Which is fine if you don't need the flexibility that kubernetes offers anyway.
And while I do appreciate kubernetes for all of its strengths, going back towards the point of this OP, I do believe it's a bit of a security nightmare. For example, the kubernetes operator pattern often just requires this application to run in your cluster with a ClusterRole granting it read and potentially write levels of access to ConfigMaps and Secrets around your cluster. Often without much in the ways of a method of telling the kubernetes API server what namespaces it is allowed to view those objects in. Worth mentioning though that secrets management is something that Docker Swarm also handles fairly poorly, and honestly isn't much of a security regression from the normal "deploy a VM that has all your decrypted secrets sitting on the system" anyway.
I wanted to eat some food so I went to the meat packing factory, hence disappointed I went to a drive through and bought burger.
I've had decent luck in the past with the K3s distribution, which is a bit cut down (but certified) Kubernetes: https://k3s.io/ Others might also mention K0s, MicroK8s or others - there's lots of options there.
It also integrates nicely with Portainer (aside from occasional Traefik ingress weirdness sometimes), which I already use for Swarm and would suggest to anyone that wants a nice web based UI for either of the technologies: https://www.portainer.io/
But even so, I still run Docker Swarm for most of my private stuff as well and it's a breeze. For my needs, it has just the right amount of abstractions: stacks with services and replicas that use networks and can have some storage in the form of volumes or bind mounts. Configuration in the form of environment variables and/or mounted files (or secrets), some deployment constraints and dependencies sometimes, some health checks and restart policies, as well as resource limits.
If I need a mail server, then I just have a container that binds to the ports (even low port numbers) that I need and configure it. If I need a web server, then I can just run Apache/Nginx/Caddy and use more or less 1:1 configuration files that I'd use when setting up either outside of containers, but with the added benefit of being able to refer to other apps by their service names (or aliases, if they have underscores in the names, which sometimes isn't liked).
At a certain scale, it's dead simple to use - no need for PVs and PVCs, no need for Ingress and Service abstractions, or lots and lots of templating that Helm charts would have (although those are nice in other ways). Unfortunately, Swarm isn't awfully good for job prospects, which I have to admit. Then again, one can sort of say the same about Hashicorp Nomad, yet it's also alive and kicking.
I’m honestly curious about Kubernetes and I want to learn more but wow, it really is hard to grok.
What does it mean that k3s is “certified”? Should I care? What’s the tradeoff?
When you mentioned Traefik my eyes glazed over.
Is there a picture of how all this fits together? One not created by a consultant selling their services?
The biggest thing to understand about Kubernetes is that you cannot understand it as a normal piece of software. You don’t directly control it, but rather, define the conditions under which a collection of automations respond to things going on. This is how it self heals. The whole system resembles more of a garden, where you are providing the conditions in which the plant grows, but you are not directly causing the plant to grow.
For example, if you understand a pod, then the next step is understanding how to keep a pod running, how to detect failures, what go do if it does fail, and how to change out one set of pods for the next. This would be a deployment.
You build your understanding by understanding more independent automations. For example, after understanding a deployment, you can look at how horizontal pod autoscalers work. An HPA knows nothing about a deployment and its internal state, yet will inform the deployment how many pods it should be running based on the metrics it sets. The HPA and the deployment scheduler otherwise work independently, and you can indeed write something that functions as an alternate HPA, and Kubernetes will do it.
There’s also an addon for cluster autoscaling. The cluster autoscaling knows nothing of deployments or HPA. It only checks pods that are being requested, but cannot be scheduled due to constraints. It then add nodes so that pods can be scheduled.
These three different processes operate independently, and don’t share state. They each watch for a limited set of signals and take limited set of actions. If any one of these processes goes down, the system can still limp along. Together, they create a much more complex behavior out of much simpler behaviors.
I’m sure it isn’t and I changed the emphasis to make that line more about the slope of the learning curve.
Thanks for this explanation. The inability to grok the internals is really hanging me up. I’ll embrace the simpler “distributions” as a way to understand the higher level constructs.
Does it also make sense to ignore the fact that this is built on Linux’s containers and namespaces?
Based on my very limited understanding Kubernetes is basically a “cloud operating system”. This is why the underlying components like networking and container managers can be swapped out. But I still don’t grok how the subsystems fit together.
The Linux container/namespace is useful in the sense that operational characteristics matter, and many troubleshooting can end up going deeper and deeper in the stack. But as far as understanding how to _design_ things, knowing how the pure abstractions are composed together lets you be able to apply it to many systems, whether it is Kubernetes or something else.
I suggest reading through the first few chapters of Roy Fielding’s PhD dissertation. He is known better for REST, but his dissertation was actually about architectural elements, properties, and constraints, and then applying to hypermedia as an engine of state.
Once you know how to reason through computing architecture, you can pick apart any architecture, framework, or paradigm, and know how to work with it, and when you fight against it. You don’t grok a system by finding out how specific parts fit with each other, but rather, how things are consequences of the architectural elements, properties, and constraints. Then, the specific way things work (and don’t work!) together pops out.
And don’t be afraid to dive into the messiness behind an abstraction. Knowing and being willing to do that can set you apart in your career.
I think that maybe I should have linked directly to the K3s GitHub, which goes into detail about what it is, what the components are, as well as links to more information about testing: https://github.com/k3s-io/k3s/
Kubernetes is a large and complex system, but doing end to end tests aims to ensure that various implementations work correctly, they have pretty in depth docs on all of this, including conformance tests in particular: https://github.com/kubernetes/community/blob/master/contribu...
So, if a distribution passes all of these, then it's likely that it will give you fewer unpleasant surprises. Here's a list: https://www.cncf.io/certification/software-conformance/
> When you mentioned Traefik my eyes glazed over.
Well, you typically do need an ingress controller that will allow exposing services to the outside and direct network traffic.
Traefik is liked by some, less so by others, but generally works okay - except that its documentation (for K3s in particular) isn't as abundant as it would be when using Nginx. For example, if you want to use a custom SSL/TLS certificate (without ACME) for all domains by default, figuring that out might take a while.
That said, if you don't like it, technically you could swap it for something else, however I personally had a cluster hang once while doing that due to some cleanup action getting stuck.
There are plenty of options: https://kubernetes.io/docs/concepts/services-networking/ingr...
> Is there a picture of how all this fits together? One not created by a consultant selling their services?
I actually found their diagrams to be surprisingly sane: https://docs.k3s.io/architecture
However, do be warned that there are many different ways how to build clusters that might have significant differences.
K3s is just one of the Kubernetes distros - that attempt to provide an opinionated view on how to mix and match the components out there to end up with something that will let you use Kubernetes and have things actually work as expected at the end of the day.
A bit like how we have OS distros like Debian, Ubuntu, RHEL and so on.
It turns out complex tools for complex tasks require some skilled operators. Yes kubernetes is overkill for your home lab, no it’s not overkill for enterprise-grade AWS orchestration with hundreds of apps and millions of users. News at 11.
If you are in a situation where you want to follow a decoupled/microservices development pattern (ie. split component ownership across dev teams) as well as manage scalability during high load periods, K8s is hard to beat.
K8s is a useful tool, but can be overkill in a number of situations. Just like a hammer, you can't use it to solve for every usecase.
P.S. F-35s use K8s clusters (or might be K3s) to deploy their EWS.
1. https://www.theregister.com/2020/06/03/kubernetes_b_21_bombe...
I've spent my entire career being on-call for web servers and Elastic Beanstalk is not enough. Deployment failures, deployment speed, deployment safety, upgrades, local testing... There is a reason AWS themselves is pushing EKS and not Elastic Beanstalk, and it's not because Kubernetes is trendy, it's because it's incredibly useful.
Kubernetes was built with no program for how to do important things. Every important thing in Kubernetes is post-hoc, poorly bolted-on patch instead of real solution.
Take for instance upgrades. Don't like upgrades? -- Try downgrades? -- Oh, what do I hear, you seem to not like that either... So, how about user management? -- Also no? -- Well, how about backups and recovery? -- What, again?
Kubernetes solved some easy problems quickly with no program for how to solve hard problems. Security is one of those hard problems Kubernetes has no idea what to do about. The reason? -- Kubernetes, on its own, provides very little. It doesn't know how to do networking -- you need a CNI for it. But there's no good interface in Kubernetes to allow the admin to somehow control in general how the cluster is supposed to behave. Maybe some container somewhere will do something tricky, and will punch a hole in whatever firewall you though you've built around it. And you cannot make sure upfront that's not going to happen. There isn't a single way to punch that kind of hole... and the problem is, often time simply to make things work Kubernetes users will punch those holes. Maybe because they don't know how to do things better, or maybe because there isn't really a better way, or the better way is so tedious and convoluted that nobody will really follow it...
It's unreasonable to expect anyone to be able to deal with this kind of complexity. We need to be able to manage complexity, but Kubernetes gives us no tools to do it. It's the same type of problem as Dijkstra described with "goto" -- it's the tools fault. It creates so much complexity that humans cannot cope. Not only humans, computer programs cannot cope -- too many things to check, to many possible problems to protect yourself against.
Upgrades and downgrades work fantastically. Putting "a firewall in front" is trivial - just... put a firewall in front. I don't know what "punch a hole" means, but if you mean "bind a port", yes, there are indeed a lot of ways to do that (on linux, windows, and osx!), and kubernetes provides excellent, organized, standardized and controllable ways to do that.
Kubernetes does not create complexity in my experience. It's the other way around - developers who have never seen the complexity in the world complain about the complexity embodied in the tools. They think "you're not gonna need it", despite an entire industry of operators saying "actually yes, we do need this, and badly".
I've seen developers who've never built a website in their life complain about React, and it sounds exactly the same as this. You might not need these tools, but they're not unneeded. If you can make a simpler and more powerful tool, please, go ahead, you'll be rich and famous, and everyone from the US Navy to telecoms to where I work will celebrate you.
Aaaaaahahhaahhahaha....
.....wait you were serious?
Just one example off the top of my head: ClusterRole vs Role. Two kinds of role. Difference? One applies to a namespace, the other applies to all namespaces, cluster resources, and non-resources. Why not just make "Role" deal with all of them based on metadata or annotations? Because fuck it, that's why. Oh, and permissions cannot include a "deny" section, they can only "allow". Why? Fuck it, that's why.
You can create a secret, but only in one namespace, and there's no way to automatically share it with multiple namespaces. Why? Fuck it, that's why.
K8s is a shitshow of both complexity and a lack of functionality. It's a design-by-committee atrocity that is so bad it created an entire ecosystem to fill in all the bullshit it doesn't do.
What namespaces do allow for, from a security perspective, is for various controls (e.g. RBAC, Network Policy, Pod Security Admission) to be applied to a given application or group of applications (depending on how you've organized your namespaces) in a relatively controlled fashion.
(Databases, of course, always like to eat everything because of poorly-optimized statements, but they've never really been good fits for K8s or containers anyway.)
Some might say that this isn't webscale, but Stack Overflow runs one of the most popular tech websites in the world.. on only nine web servers: https://www.datacenterdynamics.com/en/news/stack-overflow-st... and LiChess is similar: https://lichess.org/costs
If you don't care about high-availability, then yeah. One thing I've learned in my career is that delivering reliable enterprise-grade service is anything but trivial. The level of complexity of any modern stance is high enough that there will inevitably be sharp-edged corner cases which aren't simple or cheap to deal with.
The article about stack overflow mentions this little tidbit.
> That application handles around 6,000 requests per second, and around 2 billion requests per month.
To be able to handle loads of 1M requests per second, your going to need more than one server dispensing results. Especially since only so much data can be stored in memory without running into performance issues.
That said, these kinds of tools are an unique use case. Not all programs need them.
K8s is a way for you to drop a large number of such pieces of software onto a small number of such boxes, and keep the configuration and lifecycle of each of those deployments independent.
Kubernetes is not primarily a scalability tool it is an availability tool.
Also, the beancounters don't like on-prem as much anymore due to the upfront cost outlay.
I've been working with a number of major enterprise accounts and they've all been adopting managed K8s offerings for this very reason.
You can find k8s clusters without auth using Censys as they specifically try to access things like pod lists, and from spinning through lists of those they tend to be obvious honeypots (e.g. lots of ports that wouldn't usually go together being exposed), although I have seen the odd cluster that looks a bit more realistic, unlike Aqua I didn't try accessing it directly as that could be dodgy legally :D
If anyone wants to look at the list, Censys handily has a search query for "kubernetes clusters where the pod list is exposed without authentication" https://search.censys.io/search?resource=hosts&q=services.ku...
With that said, I agree that secrets management on k8s is pretty bad (unless you use an external service like Hashicorp Vault, a frequent pattern that I run into with k8s). Even with RBAC and secrets in a namespace being inaccessible from other namespaces, the moment someone has permissions to read secrets from a namespace, its essentially game over for you, as noted in this paragraph.
> Another potential attack vector within a Kubernetes cluster is accessing internal services. ... In cases where authentication is required, the credentials can typically be found within the Kubernetes secrets space.
[1] https://docs.k3s.io/advanced#red-hat-enterprise-linux--cento...
trusted zone is used by overlay networks or node internal stuff (or if you manually put your interface into it)
And the range of things you might want to expose... oh, god only knows. Think about such trivial stuff as FTP or iSCSI -- kinda useful, right? -- but Kubernetes rules for port exposure are idiotic, because you cannot expose ranges. And quite a few of popular Internet protocols assume they can allocate whatever port they want and start talking on it. And then you start fighting Kubernetes... and you'll win, but it will be some custom security solution... with the obvious problem of it being a custom security solution.
Now, add to this that Kubernetes will typically mess with your iptables and maybe also arptables, and if your firewall is based on eg. iptables, then you might be out of luck using it with Kubernetes, and again, will end up rolling something "special".
Obviously, you might also want, say, DNS to work inside your cluster, but that's a separate can of worms.
What is not clear in the blogs is how did they actually accessed the API server without a valid service token. Secondly, they index alot on IPs, that implied these are k8s cluster that are not managed (EKS, GKE etc).
Is my assumption correct?
You can make that mis-configuration with managed distros, which are generally exposed to the Internet by default.
The entire "CIS Hardening guide" will give you step by step tutorials on how to solve all of this, you just need to bother looking it up.
Look, you need a trained sysadmin anyway, no matter where you deploy. See for instance all the ubuntu machines out there that never have seen the admin update it.
Or all the terraform scripts that simply get copy pasted by orgs.
While I do agree, this is a problem, there are security conscious distributions if kubernetes out there like RKE2 or "rancher government" to make things much more secure.
This is actually my job. If you need someone how to secure your cluster and make sure stuff like this is very unlikely to happen, you can find the info in my profile. But what it comes down to is that if your DEVs have direct cluster access and you have no monitoring and alerting setup, you are kind of operating on the same level as copying and pasting stackoverflow answers and pushing them directly to prod without code review.
You probably know about resolv.conf, but you might not know about "search domains". A feature that should've never existed, but was added because "why not?". If you don't know what it does, in first approximation, it creates a domain alias. This is sometimes used to save some typing when using DNS for service discovery in larger clusters in a very hierarchical way (when names start to look like a.b.c.d.e.f.g...com, it's tempting to alias them to something shorter).
So, long story short. Ubuntu LTS had a bug (actually a whole bunch of them) in the same script that was responsible for generation of this file (it's a bad idea to have this file to begin with... but hey, who cares, right?). So, one aspect of the bug was that a variable name "DOMAINS" was used instead of its value. And that became a search address in resolv.conf.
Now, this search works by appending the search path to the domain name, if DNS server tells you it cannot find it. I had no idea DOMAINS was possible to register as a valid domain name. I mean, it should've been a TLD to be discoverable globally, right? -- Well, it's still a mystery to me, but, for some reason domains of the form a.b.c.DOMAINS are resolved to some AWS EC2 instance...
And, the punch line: if you screw up CoreDNS configuration somehow, or something else goes wrong with service discovery inside your Kubernetes cluster on unpatched Ubuntu -- there's a good chance your traffic will go to that EC2 instance. Who's on the other side -- I don't know. I didn't send anything important that way and discovered it because something like Spark operator didn't work... but it could've been much worse.
Oh, btw, I believe resolv.conf is used by different CNIs. Maybe Calico and Flanel. I cannot remember atm. why it was used... or maybe it's some kubelet setting.
Anyways. Larger point is... large Kubernetes clusters are virtually impossible to systematically audit. Too many things can do too many things. Too much trickery goes on in all sorts of system components that deal with networking. It's also impossible to track who does what and who owns what because there's no user identity.
---
I believe that, again, industry adopted an absolute horror show of a program / environment not because it was a great tool, but because "organically" developers started to add "features", creating something that's hard to resist, because, superficially, it creates a lot of value with minimal effort -- but it also lays a trap that is not immediately obvious. And those who find themselves trapped, instead of running away, star adding more "features" to fight the trap. But, in this foolhardy attempt, they only make the trap stronger.
The kubelet will merge the hosts resolver config with the cluster config to generate a resolv.conf file for the pods it runs, resulting in the issue you described. This can cause other annoying dns issues. It has nothing to do with CoreDNS. The only way I've found to disable this kubelet behavior is to start the kubelet with `--resolv-conf=/dev/null`
Don't be skeptical. It just sounds like you think you know better, when you actually have no clue.
> resolved to some AWS EC2 instance
How is this a Kubernetes problem, and not an Ubuntu/AWS issue?
rm -rf /
Looks like Linux is no good, back to the drawing board, time to redo the while cloud everybody.