Hacker News Reader

Comment by 0xbadcafebee | Hacker News Reader

For those unaware: Kubernetes has long had horrible default security. RBAC isn't even enabled by default. Nearly every cluster I've seen allows regular users and applications to breakout into the host nodes by default. Network ACLs, if they are ever used, almost never block either intra-cluster or extra-cluster attacks. Unencrypted API ports are standard. The distinctions between "cluster-level" resources and "everything else" unnecessarily complicates creating fine-grained rules. Default access even with RBAC is too wide-open. I won't even get into ABAC...

Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default.

I don't think it has ever been easier to take over cloud computing environments just by exploiting a dinky web app.

View on HN

RBAC is now enabled by default. But I wonder how many of these clusters that are exploitable are Openshift for example. Openshift had secure by default settings from as long as I can remember. Non-root pods as default, api-server and even metric collection ports are protected by authentication.

Openshift also had much strict RBAC settings from very beginning(like even in Openshift 3.x days, when k8s had no RBAC by default and you had to opt-in).

What I don't like about standard kubernetes (kubeadm) is how hard it is to configure pods being secure by default, so developers couldn't just mount host directories into the pods. You would need complicated configs for that and nobody does that.

For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it.

The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.

I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things.

I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking.

One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it.

There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it.

https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about.

It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.

yeah if you give a user namespace edit access, then PSA is not going to help you (apart from that default config option you listed, but then even that is tricky as you need to know the namespaces to exempt like kube-system and others)

Kubeadm is a pretty barebones option, which I think needs work to scale (you're on the hook to add all the bells and whistles to create a full k8s environment)

There's validating admission policy coming along (https://kubernetes.io/docs/reference/access-authn-authz/vali...) beta in 1.28, but you still need the policies to apply.

Really for a more "batteries included" kind of cluster, you'd need one of the larger distributions that tries to do more out of the box. Obviously OpenShift is the ultimate example of that (for on-prem) but it massively increases the complexity of the environment.

> Basically, the geniuses at Google decided to make clusters insecure by default, because being an admin is, like, hard, man. So guess what? People deployed clusters that were insecure by default.

There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant.

It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.

Also, it doesn’t help that the kubernetes documentation is awful.

I remember my time working with kubernetes, it’s a patchwork of stuff… it does require time and a fairly high pain threshold to properly master it.

Well, there are two opposing approaches: "up and running" defaults or production defaults.

You choose "up-and-running" defaults if you want people to be able to immediately run your software. So this means no encryption, no auth (or "the first user is the admin" kind of auth), storing your state in a bundled SQLite DB, etc. This helps with growth, because random devs can download the container image, run it and check out your features in five minutes. And then run it in production.

With production defaults everything is locked down. Go issue some certificates and make sure your OS (or JRE) trusts them, because your nodes won't talk to each other without TLS. No one has any rights to anything by default, can't even log in until you go and edit authN/Z configuration. The container will kill itself if the env var for the database password is still "change_this!". It's a pain to deploy this, but a pleasure to run.

Ideally, all products should come with two sets of defaults. Your AIO quickstart image should be easy to run. Your production-grade image or the configs that come with the distribution should be locked down tighter than Fort Knox.

K8s is a piece of infrastructure that doesn't really benefit from an "up and running" approach, but it has this kind of defaults for some reason instead of having every default policy resolve to "not allowed".

OpenBSD is an existence proof of a third route:

Carefully manage complexity, and write a good manual.

OpenBSD is about 22M lines of code; kubernetes is 58M, so the code bases are of comparable complexity (OpenBSD uses many more programming languages than kubernetes does).

I'd argue that the base OpenBSD install provides far more utility than Kubernetes, but market share numbers disagree.

Sources:

https://openhub.net/p/kubernetes https://www.reddit.com/r/openbsd/comments/d9a8c9/how_many_li...

Edit: Using open hub for both says OpenBSD is 75M lines:

https://openhub.net/p/openbsd

I don't follow. OpenBSD is an OS, k8s is a set of services running on an OS.

Both are things deployed to host network visible services.

> or "the first user is the admin" kind of auth

My mind immediately went to Eddie Izzard’s “do you have a flag” bit.

> There is a lot to unpack here, but while I agree with the main point (kubernetes defauls suck) the main problem I’ve seen all across the industry is that people expect to be able to install stuff around and that to be magically secure, functional and performant.

> It just doesn’t work like that. Systems administration is its own discipline and you can’t expect to follow some random tutorial off the internet and be ready for production.

Disagree. I can install an app to my Sandstorm instance and reasonably expect it to be secure, functional, and performant (inasmuch as the app is performant anyway). The tools we use make things difficult, but I don't think there's as much irreducible complexity here as you're implying. We can and should build better tools.

Sandstorm is a very different category of system from k8s.

I had the same feeling, but frankly I didn’t bother replying because somebody putting sandstorm and kubernetes on the same level shows their complete lack of any understanding whatsoever.

Kinda proves my point: some random amateur sysadmin thinking their experience with sandstorm is enough to put kubernetes in production.

But thanks for replying, anyway.

I worked with K8s professionally for several years but thanks for assuming I have no idea what I'm talking about.

Yes, they're obviously different classes of systems, but not in the ways that matter for this discussion. If the question is whether or not you can design a hosting platform with secure defaults and managed/tightly-scoped communication between applications, the answer is plainly yes. Is Sandstorm as scalable and fault tolerant as Kubernetes? Obviously not.

Out of curiosity, which bits of the k8s documentation (if there are specific bits) did you find lacking? If it's security related stuff, I can open some issues to see if things can get improved :)

K8s documentation falls essentially in two categories: either reference page for k8s components (like the kubelet) that lists all the options woth a very very very brief description … or conceptual tutorial-like that explain some concepts, but without much details.

Nothing in the middle.

There is little documentation on how to put the various pieces together. That’s all on various random webpages outside of the main kubernetes.io website.

The setup tutorial only describes some basic kubeadm-based setup. There’s a lot to improve, if you ask me.

Horrible scaling to tiny clusters as well. I'm glad they kept the design pure though. Unencrypted means you get to choose the encryption. Redis didn't have built in encrypted tcp servers for a long time either. When they added it there was a good discussion about the overhead: https://github.com/redis/redis/issues/7595

Whilst Kubernetes defaults are indeed not great, they aren't that relevant unless you're building your clusters from scratch instead of using a k8s distribution, and I don't think I've seen anyone do that in a long time.

RBAC is enabled by default on just about every distribution I've seen (with the notable exception of microk8s)

Lack of default admission control to prevent someone with pod creation access breakout to the host is indeed pretty common

Unencrypted APIs hasn't really been a problem in years, the insecure API port and read-only Kubelet were a thing but they've been disabled for quite a few versions now.

I'd definitely agree that early k8s versions had very bad security, but things have been improving gradually over the versions :)

Well first of all, there are still thousands of clusters running with the old insecure defaults, so that's bad.

Both GKE and EKS currently have insecure defaults. There's a half dozen ways to take over the entire cluster with the default settings.

The unencrypted API port is only this year being deprecated on GKE, I don't know about EKS.

Since you can only grant permissions, not deny, most people end up granting too many permissions, defeating the purpose of least privilege.

Default permissions for users on GKE/EKS are way too wide open. You can do things you just shouldn't need to do as a regular dev.

You shouldn't be able to just deploy a helm chart to a random cluster and then all of a sudden there's an unauthenticated public service on the internet. But that's what every single distribution of K8s gives users by default. Regardless of whether you think that's fine or not, this is how companies get exposed on a regular basis, so it is a real-world persistent security problem that is perpetuated by the defaults.

People not upgrading is indeed a problem but not one the distributions or k8s can particularly address, unless they force upgrade people which with deprecation/removal of APIs doesn't tend to be popular :)

The support lifecycle is pretty short, compared to general enterprise software, so unless orgs are happy running unsupported versions of k8s they're on 1.23 for EKS or 1.24 for GKE by this point.

I wouldn't have said that EKS, in particular, had very bad defaults, IME it's the best of the managed distributions for security. Any references for 1/2 dozen ways of taking over a cluster from defaults settings? I'd genuinely be interested as I collect information about Kubernetes security, so I'm always on the look-out for more references.

for unencrypted API are you thinking of the Insecure API port or read-only Kubelet? For the insecure API port I don't think that's been a thing for a really long time on managed distros. The only time I remember seeing it was on ACS (the predecessor to AKS) where it was exposed to the container network. read-only Kubelet was more of a thing as that was on worker nodes, but it was less risky.

I've seen non-tech corporations adopting Kubernetes without realizing that it's just one part of Google's infrastructure, not the entire operations side. There's an entirely separate chunk of infrastructure that provides the kind of security you mention.

K8s is a "do one thing and do it well" that does workload orchestration. Google's BeyondCorp system enforces a zero-trust policy. It's not left to each part of the infrastructure to independently implement.

Well, Google uses Borg internally which isn't exactly Kubernetes. But yes, the idea is that for it to be useful as infra it has to come without a lot of magic automatically attached.

"Default access even with RBAC is too wide-open."

Not following this. Isn't a k8s cluster behind a network firewall? Only exposing 443? To me RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security. That sounds awful.

Jeff in Engineering might have deployed an app spanning vCenter as well as EKS, and to continue to allow the app to function wrote subpar ingress/egress rules.

443 is just a port. If we assume that implies HTTPS then that’s still only TLS. If I exploit the service (“securely”!) I am inside your network. I can then pivot to your management layer. Game over.

> Isn't a k8s cluster behind a network firewall?

Of course not, unless you configure it this way.

> RBAC would be useful in a specific case where you want k8 service accounts (no one uses these) to have different levels of security.

What? How comes no one uses service accounts? They're used extensively for many things.

Reply on news.ycombinator.com