Kubernetes Hardening Guidance [pdf] | Hacker News Reader

Kubernetes Hardening Guidance [pdf]

media.defense.gov

237 pointscjg60 comments

This is one of the standards and compliance guides you can use for k8s.

The other ones I'm aware of are

- CIS Benchmarks, there's coverage for Kubeadm, AKS, EKS, GKE, OpenShift and some others. This is a compliance guide focused on just k8s

- DISA STIG for Kubernetes, another compliance guide, they don't mention which distribution but it's kubeadm from looking at the paths mentioned.

- PCI Guidance for containers and container orchestration, this one is recent, it's a generic guidance targeted at container environments (docker, k8s etc) for PCI in-scope organizations but TBH it should work for most places (if that one's of interest, some more info https://raesene.github.io/)

Some more details on these https://www.container-security.site/general_information/cont...

Making security guidance for k8s is kind of tricky due to the number of distros and changes between versions (https://raesene.github.io/blog/2022/09/20/Assessing-Kubernet...)

> Kubernetes, frequently abbreviated “K8s” because there are 8 letters between K and S

I'll be damned. I thought it was because the end kind of sounded like "8-es"

Same as i18n (internationalization) and l10n (localization).

a11y for accessibility also (and perhaps "ally" too?), but is a bit ironic given that screen readers wouldn't be making much sense out of that one.

And a16z. It's a pretty common silicon valley convention. I was even asked to implement this in an interview once.

ha, me too, 4-5 years ago!

I wish I had that as an interview question.

What is a16z ? A google search only returns references to Andreessen Horowitz

Not sure if you are joking but that is what it means

so you are asked to build the best ARM CPU in the world?

XML-Canonicalization is xml-c14n too!

* According to Tex Texin, the first numeronym of this kind was "S12n", the electronic mail account name given to Digital Equipment Corporation (DEC) employee Jan Scherpenhuizen by a system administrator because his surname was too long to be an account name. By 1985, colleagues who found Jan's name unpronounceable often referred to him verbally as "S12n" (ess-twelve-en).*

https://en.wikipedia.org/wiki/Numeronym

You are not alone! I didn’t know

I find the fact that the Defense dept issues stuff like this almost more interesting than the content itself. Says a lot about what keeps them up at night

The Space Force is cool, but honestly, we desperately need a Cyber Force.

https://www.cybercom.mil/About/Mission-and-Vision/ ?

https://www.nsa.gov/

Yeah but it'd be cool if we had a governmental agency that wasn't adversarial to its own citizens.

The nsa is a bit of an oddity. It has conflicting branches within it's own organization. Parts of it want to keep you (and the rest of government) secure and the other parts want backdoors into everything.

No backdoors! Secure everything! Except the stuff we want access to!

No, there are two teams. That's Joe. His job and budget is to secure everything. That's John. His job and budget are to get us access to everything. Joe and John work for the same group, but they don't really work together. Although I can see John looking at what Joe is about to publish a few days before.

Wait until you hear how wall street firms can have investments that are directly correlated to stock price and inversely correlated to stock price on the same stock!

Can you explain the wall street portion? Not following what you mean.

It's totally possible for one firm to have two traders on opposite side of a trade both working for them. Just like the NSA can hire hardening experts and breaking in experts.

Wouldn't it?

https://www.cisa.gov/

A lot of what a cyber force would do falls under DHS

The Marines created a hacking MOS and a programming MOS in the 2200 specialty I believe.

Link: https://www.marinecorpstimes.com/news/your-marine-corps/2014...

Having worked exactly in this space, this is a perfectly normal document. Architects and principal engineers aren't bothered to the point of distraction but it is definitely something that's taken seriously. And it isn't just the DoD, it's every federal agency.

You may also find the US Government Accountability Office interesting.

They do a mix of various reviews and audits of government services and projects, and publish and targeted recommendations; But to form a basis for those reviews and audits they have a position on what "good" looks like and publish that as guidance.

[1] https://www.gao.gov/

It means the defense department or its vendors are using Kubernetes, which they are. That should keep them up at night.

Yup. Apparently even their newest stealth bomber uses it internally: https://www.theregister.com/2020/06/03/kubernetes_b_21_bombe...

I used to work in a really niche engineering field. The amount of stuff I used from documents released by the US military was non-trivial. Never anything directly, but most of the time the docs were useful enough to shed deeper light and context on the trade publications and standards I was using when the history and intent behind a lot of calculations wasn't clear.

Is it possible to configure a Kubernetes cluster to run only _signed_ images? I.e., if someone has replaced a Docker in registry is should not be accepted by cluster.

You can probably use an Admission Controller with the ImagePolicyWebhook.

https://kubernetes.io/docs/reference/access-authn-authz/admi...

Came here to ask the same thing. If I can sign git commits, can we sign images or even individual layers?

Yes, see https://github.com/sigstore/cosign/blob/main/USAGE.md for one example

There's an ecosystem of policy control tools built on top of k8s' ValidatingWebhooks. Check Open Policy Agent and Kyverno.

https://www.openpolicyagent.org/

https://kyverno.io/docs/writing-policies/verify-images/

The short answer is yes. There are multiple tools that let you do this.

My personal favorite tool for this is Kubewarden[1] because its policies are web assembly. There is a specific policy just for verifying signatures[2].

[1] https://www.kubewarden.io/

[2] https://artifacthub.io/packages/kubewarden/verify-image-sign...

Being able to specify images by hash would be a simpler alternative.

Requiring signed images seems like an arbitrary place to require signatures, given that there's plenty of parts of kubernetes deployment configs that could be used to do damage and you need the whole thing authenticated. I guess a benefit of having signed images instead of content-addressed images is that they could be updated by a trusted person without needing to update any kubernetes deployments, but presumably you'd want to tell kubernetes to switch its running instances to the new images so that sounds like an incomplete solution.

If you use cri-o as the runtime along with an openshift container registry, it will actually verify signatures at the runtime layer. In addition to crio, podman and anything based on containers/image supports this too.

Really that just means a registry that sends back a header indicating it supports signatures and serves up the right signature endpoints. It's shocking this isn't more common.

But if you just want to check signatures at the cluster's point of entry, you can use an admission controller to block the pods from being created with unsigned images.

For those who are implementing these security guidelines: how do you ensure they have been correctly implemented?

Do you have any kind of static check program that can check beforehand that you are going to deploy a hardened kubernetes cluster? Do you have a "live" checker that can verify the actual configuration of a running cluster? Does it run all the time oronce in a while? Also , if you have an automated way of verifying your configuration, which program do you use?

I only know about Chef's Inspec and the CIS profiles that are available online, but the experience wasn't extraordinary and I was wondering what is used in the wild?

Maybe you can add them to OPA? (https://www.openpolicyagent.org/)

The people that define the standards don't implement them. The people that implement them don't validate them. Government systems don't typically have any kind of system that periodically checks system configuration, though the systems might get reviewed every few years.

A lot of these things can't be meaningfully statically checked. It has to be done by a working policy apparatus. If your org can't adopt, implement, and adhere to policy then there's no amount of standards documents in the world that will help you much.

There's no way in the world to statically and automatically check if your org regularly reviews configurations, responds correctly to monitoring alerts, ensures your developers adhere to least privilege principles, and so on. It has to be policy.

So there are various commercial systems that fall under CSPM (or KSPM sometimes) that are designed to assess compliance with different standards.

My purely personal opinion on this is that it's difficult to do well as even with compliance standards automating assessment isn't always possible

For example the CIS benchmark for k8s can't say "Never use cluster-admin" as there are some legitimate use cases, so instead it says "minimize the use of cluster-admin" which can't be fully automated as a check.

To do it well, a company should come up with their own spin on applicable standards, automate where possible (either with 3rd party or internal tooling) and then manually review the things that can't be automated on a periodic bassis (either with internal resource, or consultants)

I'm implementing it right now actually and it's a combination of automated tools and someone third party actually going through and verifying all the controls.

Prior discussion: https://news.ycombinator.com/item?id=30692794

Another guide may be the CIS benchmark guide [1].

I can't attest to efficacy of this particular benchmark from defense.gov (we don't use k8s at $DAYJOB), but we've leveraged other benchmarks from CIS for various flavors of Windows/ Linux.

[1] https://www.cisecurity.org/benchmark/kubernetes

The example of appendix A is a PoLA violation (Principle of Least Authority).

It has source code in the container. Use an external build server.

Well done. From the control plane section:

>The Kubernetes API server runs on port 6443, which should be protected by a firewall to accept only expected traffic.

How are folks doing this in practice at scale? Managing ACLs for kubectl, admins, workflow systems, distributed worker nodes etc?

There's actually quite a lot.

meaning lots of ACLs on both sides of the api server, or meaning lots of better (simpler, more secure) ways to do it?

For a bare-metal cluster one simple set and forget about it solution is to port forward another external port (e.g., randomish 51203) to the internal control-plane-ip:6443 and block port scan attacks using fail2ban, DenyHosts, psad, etc. This should prevent most of the attacks.

I didn't read this, but it's really tiresome to hear about having to "harden" systems in 2022. They should be "hard" by default. If you need to soften them to make them easier to work with internally, that should be what needs a checklist and instructions.

Looking at the guide, much of the hardening is above and beyond anything any Kubernetes configuration can be expected to do. Thus there is literally no way to ship a pre-hardened Kubernetes that has to be softened with a checklist guide.

Much of it is about development practices. Kubernetes cannot scan your containers for vulnerabilities and misconfigurations for you. Kubernetes cannot ensure lease privilege practices in your containers for you. Kubernetes cannot do regular reviews of logs and configurations and security patching status for you. Kubernetes cannot monitor audit logs for you.

With all this said, it's worth taking a look at the guide. It goes far beyond suggesting a few changes to default settings. Perhaps it could have been better phrased as "Hardening the context and practices around Kubernetes" to avoid this confusion.

I think this is true, and the issue is deeper for kubernetes. A k8s cluster by itself is not particularly useful. In order to deploy real software you're going to need a bunch of other components, extensions, and software. Things like persistent storage, ingress controllers, service mesh, certificate managers, DNS, etc. All of those components require consideration from a security perspective.

Even OpenBSD ships with post-install “hardening” guidance…

https://man.openbsd.org/afterboot

> I didn't read this, but ...

Yeah, but then everybody bitches about ease of use. That other project down the street just works out of the box, while your project is mired in configuration hell.

More like we need a better Dev mode vs Production mode switch. Dev mode would be fairly insecure but would also refuse to run on the internet. Production mode would ease deploy but also "self harden".

That kind of divide ultimately falls apart. Over time, things built in dev mode rely more and more on its insecurity and production systems get pushed that way.

This is why development systems need to be as production-like as possible. Otherwise people ship boring webapps that inexplicable rely on running as root in privileged containers and expect prod to enable this.

Reply on news.ycombinator.com