Hacker News Reader

> The only new thing I’m seeing from this solution is the ability to create containers as an unprivileged user, which for most scenarios I tend to view as a non-issue. Or am I misunderstanding that?

I believe you're mislabeling it as a non-issue, it seems to be the core problem they're trying to solve:

From the blog[0]:

> Yes, we all know that containers run unprivileged processes; but creating and running the containers themselves requires root privileges at some point.

> What is the awesome sauce we all gain from this?

> Well judging by the original GitHub issue about unprivileged runc containers, the largest group of commenters is from the scientific community who are restricted to not run certain programs as root.

[0]: https://blog.jessfraz.com/post/getting-towards-real-sandbox-...

I did read the blog post, but it all seemed about running containers as unprivileged users. Being able to create is what I’m saying is a non-issue, because it only saves a single convience step of not having to upload a container built elsewhere to a cluster. And tools like Singularity have online tools (Sylabs Library, Singularity Hub) which offer remote builders for users that don’t have privileges anywhere (which is hard to find) and don’t want to build containers locally. You can then pull and run the produced container from the Singularity executable without needing root access.

An alternative is a cluster could set up a single machine that gives users sudo access only on the Singularity executable and prevent the ability to bind mount, therefore creating a relatively secure build environment on-premise. However, security matters more than conviencence for HPC clusters, so we’re more inclined to tell users “do it yourself” than give them any loophole in the system.

Singularity already solves the unprivileged running of containers. Unless binctr is doing something that provides a different kind of unprivileged access?

I’m not trying to be thick headed, but I’m just not connecting the dots on what’s so special about this.

> I’m just not connecting the dots on what’s so special about this.

It's not. binctr is not special. Singularity is not special either. They are both tools that do similar things.

singularity does partially support running without suid: https://github.com/sylabs/singularity/issues/1258

charliecloud is designed specifically not to need suid: https://github.com/hpc/charliecloud

docker will hopefully support full unprivileged use soon: https://github.com/moby/moby/pull/38050

The Docker work is a direct derivative of the rootless containers work I started more than 2 years ago (and others have been working on before and since), which is what this blog post refers to.

Singularity didn't exist at the time in any meaningful way, and suid binaries (even a small number) are completely unacceptable for the usecases I had.

Thanks for those, I’ll read through them when I get the chance!

> I’m trying to compare/contrast the goals of binctr to existing solutions that appear to do the same or similar thing(s) already, so I’m open to learning something new today!

binctr is a proof of concept that was written almost 3 years ago. These days, rootless containers are being integrated into many tools. Right now, under [1] we have some tooling which allows you to run Docker, cri-o and Kubernetes in rootless mode. Cloud Foundry's default configuration now uses rootless containers.

So really, it's not a choice between binctr and something else (binctr was a proof-of-concept) -- in fact please don't run binctr in production. But runc and quite a lot of other tooling has rootless support now.

Singularity doesn't do rootless containers, it still requires suid binaries in several cases. But I'm biased, having had some pretty bad arguments with the lead developers in the past (mainly for misrepresenting runc).

[1]: https://github.com/rootless-containers

Thanks! This helps me understand a bit better. So away from binctr, which should just be considered a reference, a rootless system isn’t just about being able to run i.e. execute a container, but also go into and modify, like using the containers package manager and fiddling about. Without needing root access to do so (due to the way these tools like yum are built to work). But also be able to use systems like Docker that typically require sudo/root for nearly everything to not need it.

For us, one of Singularity’s primary advantages is the host filesystem integration capabilities.

Does a rootless container system have any potential drawbacks when dealing with host filesystem integrations and user permissions? Or should this be considered up to the implementer’s discretion and a separate issue?

This may be thinking too much (but probably due to lack of deeper understanding) but an event I see playing out is a user on the cluster installing a rootless container system, building their own containers, runtime mounting host directories they shouldn't be that typically require root access and then modifying them. Is this a possible situation?

I often end up getting caught up in tooling and missing the bigger picture/goal, so apologies for the long back and forth :)

> a rootless system isn’t just about being able to run i.e. execute a container

This is part of it, but I want to repeat again that Singularity (and most projects that you've mentioned) only appear to not require privileges -- they use setuid binaries. That isn't the same as not requiring privileges (in my opinion -- because requiring setuid binaries means you need root to install something on the host).

Rootless containers can be used without any root access at any point -- even during container runtime installation.

> but also go into and modify, like using the containers package manager and fiddling about. Without needing root access to do so (due to the way these tools like yum are built to work). But also be able to use systems like Docker that typically require sudo/root for nearly everything to not need it.

You have "root" inside the container, but this is a user namespace so it's actually an unprivileged user on the host (and you don't need to have privileges to create a user namespace). Package managers get a bit complicated, but we have techniques to get around their lovely issues (or you can use newuidmap/newgidmap that map a range of UIDs/GIDs and then it all works without issue -- these are standard setiud programs you get with distributions and most distributions will give users mappings by default).

Docker is a bit complicated (to run as an unprivileged user or inside a container), but with some patches (which are mostly upstream from memory) you can run Docker inside a user namespace with a bit of finagling thanks to the rootless work done by myself and Akihiro.

> one of Singularity’s primary advantages is the host filesystem integration capabilities.

One of my problems with Singularity is that they keep creating new words to define old concepts -- can you explain what you mean by "host filesystem integration"? Are you talking about bind-mounts? If yes, then yes you can use them.

> This may be thinking too much (but probably due to lack of deeper understanding) but an event I see playing out is a user on the cluster installing a rootless container system, building their own containers, runtime mounting host directories they shouldn't be that typically require root access and then modifying them. Is this a possible situation?

Yes, this is precisely the long-term usecase we want to have. Right now you can run Kubernetes unprivileged (in a mode we call "usernetes"[1]) and the idea is that you will be able to login as any unprivileged user on a cluster machine and set up Kubernetes (or just run containers directly if you don't like Kubernetes). There are some complications with networking, but we're working on it.

[1]: https://github.com/rootless-containers/usernetes

> Are you talking about bind-mounts? If yes, then yes you can use them.

Yes, I’m referring to the bind mount capabilities. By default they are enabled, but Singularity’s config file allow us to disable user defined mounts and set global mounts that will always be used.

> Yes, this is precisely the long-term usecase we want to have.

I’m confused. The situation I described was a negative where a user run container bind mounts directories they shouldn’t have access to and modifying it. Like mount host /etc to container /mnt then start editing the host systems configuration files. I may be misinterpreting your statement but I don’t see how that event should be a long term goal...

> Yes, I’m referring to the bind mount capabilities.

Okay. Every existing container runtime supports bind-mounts. Having a global configuration for them is mostly a UX thing (you could make a wrapper in half an hour that does that).

> The situation I described was a negative where a user run container bind mounts directories they shouldn’t have access to and modifying it.

Oh I misunderstood your question. No, that's not possible. "Root" inside a user namespace cannot access files that the underlying user would not be able to access -- this is guaranteed by the kernel (which is why I misunderstood your question -- it's a kernel security policy which affects every process).

I thought you were asking whether it's possible to run a cluster as an unprivileged user. An unprivileged user can bind-mount whatever they want, but actual file access is blocked when you would expect.

As someone who doesn't know a lot of the nitty details of the kernel or containerizing, I appreciate you for taking the time explaining some of this.

So just to make it completely clear in my head (sorry!), when utilizing a user namespace rootless (non-setuid) container system:

* The user has complete read/write/execute to everything in the 'base' container. Anything bind-mounted in assumes the given privileges from the host system.[0] The user's context is essentially mapped to root inside the container, but not a 'true' root in the context of the host system.

* The user can create/modify/run a container without needing root privileges to communicate with a daemon or any other tooling.

* A user can install a rootless system to their owned directories on a host system and operate its services/tools without needing elevated privilege access.

[0] In the situation where a user maps a root owned directory to its corresponding directory in the container (/usr:/usr) the container root privilege access is no longer applicable. Other permissions (rwxrwxrwx user group) also apply inside the container from bind-mounts.

If the above is all true, then I think I've got it. All in all, I do think the idea of a rootless system is pretty cool. Obviously you need to be running a 3.8 or later kernel (unless backported for namespaces) and enabled in the kernel, but its cool tech. Thank you for taking the time to help me understand it better. This probably would have gone a lot faster if done in verbal conversation, so I appreciate you sticking it through :)

If you don't mind me asking, what came up in your conversations with the Sylabs team? Was it a case of "we'd like to wait and see as this is new[1], not thoroughly vetted tech", not developed in-house, or they just didn't want the tool to work that way? Those are just speculations I can come up with off the top of my head. If you don't want to talk about it, that's fine, too.

[1] Relatively speaking at the time. It's clearly not brand-spanking new.

> The user has complete read/write/execute to everything in the 'base' container. Anything bind-mounted in assumes the given privileges from the host system.[0] The user's context is essentially mapped to root inside the container, but not a 'true' root in the context of the host system.

Sure, but it's actually much simpler than this. User namespaces allow you to change the UID mapping of users -- and an unprivileged user can map themselves to "root" (however, on the host, the user is still the same -- this similar in concept to a PID namespace). However, in a user namespace, filesystem access (plus quite a lot of other things) are restricted based on your real UID/GID.

A container image (in order to be used by a user namespace) needs to be extracted such that the owner is the user that you want to execute as. If you were using "root-ful" user namespaces then you could just chown an existing image, but for rootless you need to extract it. My OCI image tool umoci[1] has supported this extraction (it's a bit more complicated than you might think) for about 2 years now.

The point is that a container image is generally just a directory (maybe it's a complicated mount hierarchy, but at the end of the day there is a directory that you can "chroot" into -- note that we actually use pivot_root but that's a different topic).

> The user can create/modify/run a container without needing root privileges to communicate with a daemon or any other tooling.

Yes. The idea is that if you are dropped on a box (with a new enough kernel) with just an unprivileged user, and none of the tools you need are installed, you can still run containers. So all aspects of the container runtime must not require privilege -- and runc has supported this mode since 2016 (I implemented it).

> If you don't mind me asking, what came up in your conversations with the Sylabs team?

The main disagreement is that they misrepresent runc on their project page[2] -- which is hosted on a .gov website no less. After that there were some further disagreements about whether rootless was something that they actually supported (they don't -- and in fact I was fairly sure I saw several vulnerabilities in their setuid helpers, but I didn't bother looking further into it). Their documentation now says that they've supported rootless containers since 2016 but I don't agree with that statement (it's also so restricted)[3].

I got (somewhat unfairly) angry with them over this whole exchange, so I don't really like discussing Singularity that much anymore. In my first talk about rootless containers, someone actually asks about Singularity[4].

[1]: https://github.com/openSUSE/umoci [2]: http://singularity.lbl.gov/archive/docs/v2-2/faq#why-cant-yo... [3]: https://www.sylabs.io/guides/3.0/admin-guide/security.html#h... [4]: https://www.youtube.com/watch?v=r6EcUyamu94