How We Built SELinux Support for Kubernetes | Hacker News Reader

How We Built SELinux Support for Kubernetes

gravitational.com

70 pointstwakefield32 comments

I've written a bunch of SELinux policy, also without being especially expert. The tooling is definitely getting better, but still there are many annoying things from the obscure file layout, the giant all-in-one everything-is-a-global policy[1], to the small annoyances like m4 in the policy files.

Of course you need to buy into the whole premise of SELinux - that in addition to simple Unix permissions you can use mandatory access control to fine-grain control everything that a process can do and get access to. Obviously this is a giant boil-the-ocean project that has consumed many good people for decades. For SVirt (secure virtualization) this was pretty successful, allowing many qemu instances to run on a single server, and confining them with SELinux so each can only access its own files even when they are all running as the same Unix user.

The linked article is a good intro.

[1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project.

> [1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project.

+1 this is sorely needed and I wasn't sure if the design would ever evolve to include this

Worth mentioning that SVirt is also used for running containers on RHEL/Fedora/CentOS etc.

I don't think that's the case, svirt is specific to libvirt + selinux-virt, whereas containers use selinux-container which are integrated into docker & al. Same principle tho

Originally all the domains had 'svirt' in the name:

https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_con...

These days containers have their own domains:

https://www.mankier.com/8/container_selinux

As you say, the principle is the same.

[edit] container_t is (or was?) an alias for svirt_lxc_net_t:

https://danwalsh.livejournal.com/79191.html

While the article shows how they build SELinux policy from scrath, they also mention udica tool[1] which generates SELinux profile based on container inspection:

# podman inspect my_container_id | udica my_container

I wonder how much would the result be different in their case or how much time would they save by having the first version of the policy being generated this way instead of writing it from scratch.

[1] https://github.com/containers/udica

What's the point of SELinux support in a k8s environment, where services are already sandboxed by the container runtime?

It's unsafe to run untrusted workloads in the host kernel using only namespaces - SELinux does not change this, no matter how much Red Hat wants it to be true. The kernel has a massive attack surface. For untrusted workloads, you need virtualization like gVisor or Kata/Firecracker which isolates workloads from the host kernel.

SELinux can mitigate some logic bugs in the container runtime, yes, but it doesn't make your containers secure.

Also, the title of this submission on HN is inaccurate - Gravitational did not build SELinux support for k8s. Red Hat did, a long time ago. They added support for SELinux to their own product, Gravity.

My naive understanding is that containers do not provide complete security.

Also, I could see giving a user shell access within a container and still restricting what they do with SELinux.

Defense in depth. Running untrusted code in a kubernetes cluster is risky even with the sandbox. SElinux adds another layer of protection

I would go further and say that SELinux (and AppArmor) aren't defense in depth but actually a necessary part of the tooling to implement sandboxing of containers.

SELinux would have prevented a number of container breakout issues.

That said, as someone one time worked on an SELinux issue in Docker and then became the defacto "expert"... it's a terrible mess, often coming with incompatible policy changes tied to kernel changes and policies which are stacked on top of each other like a house of cards that are difficult to tease apart for other distros.

Because MAC (ie. SELinux and AppArmor) is part of the tooling the container runtime needs to sandbox services. Without SELinux you end up with the awkward situation where services are running as real users on your system and there's no real way to confine them beyond that.

OpenShift is the only Kubernetes distribution with SELinux in enforcing mode out of the box (that I know of) and this has stopped attacks in the past.

The point is that containers are actually just running PIDS, if you were to escape the process, you would have access to the host without security measures like selinux. You can also use selinux stop things like interprocess communication.

Something I notice in regards to selinux, is many people don't understand that you are not just applying context to files, but also PID's. You can see the context of a PID by doing 'ps -eZ', similar to how you would with files. This allows you to create a finer degree of process seperation.

You are right though, RedHat has been doing this with OpenShift since 2011, and it is a major part of their security model. RedHat's current push is the deployment of rootless containers, which addresses similar concerncs.

An off-topic but somewhat related question. What is a good way to whitelist which applications can access the network on my desktop linux system? (Currently running Arch, but I distro-hop quite often). Years ago when I had a MacBook, I used a program called Little Snitch. I see that there is a similar application for Linux called OpenSnitch, but I’m hoping there is a better way to utilize the built in features of Linux rather than introducing a 3rd party kernel module. Are SELinux / AppArmor the answer? Cgroups? Something else?

Selinux can add labels to network packets which can be matched with nftables & al. Its a bit complicated tho. Since you need to add and create labels.

You can also match by owner in iptables. So you can have a network restricted user and sudo to it.

SELinux was a good way of securing linux machines with many services sharing things like file systems and binaries.

k8s solves this in a different way. You can write applications that are completely stateless when spun up on a k8s node and can be limited to a read only file system. Your container can be a scratch container with no other executables binaries.

I don't know what their product is, but i seriously doubt mixing these two concepts is the right thing to do.

I think you need to look at how it really works under the hood if you believe that. RHEL runs containers with a few layers around them, one of which is SELinux (example of an exploit being blocked: https://www.redhat.com/en/blog/latest-container-exploit-runc...)

I'm not sure it's fair to say the advantages of selinux are solely around sharing services and filesystems on a host. I believe selinux also allows a more expressive policy on how a running piece of software can interact with the kernel than say seccomp can, which is probably what you're container runtime is using to limit kernel access. And unless you're configuring seccomp, you're probably picking up some default policy designed to work on as many systems as possible, which sort of violated least privilege.

I'm also not sure I would consider this a totally solved problem within kubernetes. In supporting gravity, our kubernetes distribution, we used to hard-disable all privileged containers. This become a common point of friction, since many pieces of common kubernetes software seemed to just use privileged containers instead of properly listing and setting needed capabilities. Although this is a separate issue from selinux, I'm just pointing out that many kubernetes clusters may not be as isolated between running processes as expected.

And to answer what the product is, gravity is a kubernetes distribution, that in this use case allows for packaging a kubernetes based application together with the runtime, for a unified installer for the application + runtime. Think something like packaging up a SAAS application, and being able to install a copy within a banks network who doesn't want their systems talking to the internet while abstracting away the kubernetes portions.

The problem we were running into, is that end user would keep trying to install the application onto selinux enabled hosts, which would break the install or runtime. And whether you believe in selinux or not, it causes significant friction to try and tell your customers we got this, just turn off that security feature in production.

Given the additional context “we need out k8s deployment to work for customers who use selinux on their nodes”, the motivation to do this makes more sense. The majority of k8s users likely don’t share their clusters with untrusted/low trust users so it seems unlikely to help them much.

Not to dunk on the work you’ve done: it’s still awesome. Just want to point out the disconnect from users wondering how this would help them.

We do try and point this out in the article itself, but apparently failed to convey the total context by trying not to spend a lot of time explaining gravity, which for the particular article shouldn't be that important.

> The problem we had was that Gravity was never designed to run on SELinux-enabled hosts and our customers kept randomly breaking their installations when trying to run Gravity with SELinux enabled.

I think this misrepresents the use case. Most customers that require k8s on SELinux don't share their clusters with untrusted users. It provides much stronger isolation between containers in the case that one is compromised, reducing the scope of damage when a system is successfully attacked.

In other words, they don't assume that a trusted container/user stays "trusted" operationally.

Their product is (or at least was, when I used it back in late 2018) bootstrapping a K8S cluster on bare metal with "one command". Obviously this is an extremely aspirational claim.

NSA made SELinux. My theory is that they made it so hard to use so nobody would use it /s

Redhat helped the NSA make SELinux substantially easier for more people to use. Disclaimer: It's still not super easy to use. The implementation at the NSA did not have the concept of "unconfined" anything and they use MLS policies. At least, that was the intent. This may have changed with time and perhaps by now they are using Redhat's version of SELinux.

Tools write selinux policies, humans write apparmor policies .

apparmor is dead at this point right. selinux is the standard

could you explain why you think so? We are are Debian/Ubuntu shop and it looks like it's all AppArmor.

AppArmor is used by debian/ubuntu, while selinux is used by Redhat and Android. Don't think any of them is more dead than the other. The refpol used by redhat isn't very restrictive/complete and no one except for big corps are using it. Apparmor is barely looked at too in my experience.

In fact AppArmor was only "recently" enabled by default in Debian starting with Debian Buster: https://www.debian.org/releases/stable/amd64/release-notes/c...

apparmor is also the default linux security module for suse

Both tools can be managed by both tools and humans. SELinux is for Redhat distros and operates on inodes. AppArmor is for the Ubuntu distros and operates on files/dirs. Each have pros and cons, too long to cover here at HN.

Reply on news.ycombinator.com