The Quest to Secure chown and symlinks | Hacker News Reader

The Quest to Secure chown and symlinks

buildkite.com

59 pointsjuanfatas27 comments

This just cements my conviction that file systems not having transactional operations is a huge omission nowadays. It really is time to start having file systems that are not just huge mutable spaces, and be more like proper ACID databases.

I hope somebody is working on it because as things are going in the last years, I'd be retired before I have the time for it.

How does that solve the problem of links pointing places where you don’t expect them to, or any of the other issues in this article?

The problem here is trying to cross a security boundary where your only tool is shell scripting. That’s just basically impossible to do securely.

Use a real programming language, follow the rules required to make it secure and do all the checks you need to.

It would solve it by disallowing changing of the underlying path from a symlink to a file (and vice versa) while a transaction to do a `chown` is still underway.

Though that would require much more than just ACID semantics but also proper user / jail isolation.

  (cd "$path" && [ "$(pwd -P)" = "$path" ] && chown -R buildkite-agent:buildkite-agent .)

the real question though is why they're trusting just Docker alone to isolate customers; if they want the jobs to effectively be a single user to the system, they can even use unprivileged user namespaces?

This stack is run by a single customer on trusted code isolated in their own AWS env. (I wrote it originally 6-7 years back)

There are radically better isolation strategies now. Firecracker and/or Sysbox hardened docker containers is one I’ve recently implemented.

When all you have is a hammer...

I have a small project in this approximate area:

https://www.kylheku.com/cgit/safepath/about/

safepath is a function which tries to analyze whether a path is safe to use. Roughly that means that it doesn't resolve in some way that can be controlled by another (non-root) user.

A something similar to this is in TXR Lisp under the name path-components-safe:

https://www.kylheku.com/cgit/txr/tree/stdlib/path-test.tl?hl...

This seems to still be vulnerable to TOCTOU?

Tho, to be fair, it's probably an improvement by narrowing how simple it is to exploit relative to doing nothing.

I'm not going to sit there claiming it's not vulnerable to anything.

But, assuming it does what it's supposed to, the inherent semantics of the check is supposed to eliminate TOCtoTOU.

The reason is that if we validate, from left to right, that no part of the path is under the control of adversaries, then nothing can happen to that path between the time of check and time of use.

(At least nothing that isn't self-inflicted, which would be outside of the scope. If root checks a path for safety, but root itself has a parallel task going on which changes the permissions or symbolic links in ways that affect the security status of that path, we can regard that as root's self-inflicted problem. That's nearly the same as the ever-present threat that a careless sysadmin might "chmod 777" an sensitive file.)

In fact, development of this function was motivated by exploring the question: can we do security checks on a path in such a way that if the answer is affirmative, it continues to tell the truth moments later, when the same path is accessed?

My objection would be that one (or more) path component(s) could come under the control of an adversary in between the check and the use. Granted it might take a weird set of circumstances -- e.g. a deployment process running in parallel doing odd things with permissions[0] -- but the fundamental problem seems to be essentially unsolvable without kernel support.

It's definitely a great improvement, though.

[0] Which would arguably be a bug with the other process, but this is the world we live in :/

Right. Basically the deployment would have to happen in a secret tree: a directory with a hard to guess name (e.g. 256 bit random hex string) in a root-owned directory that is not readable to anyone but root. Only when the deployment is finalized (all chown and chmod operations have been done) is that tree then renamed to its deployment location.

Whenever root creates an object owned by root, which is then chown-ed to non-root, if an adversary can point a root process at that object, that could subvert safepath.

Just a little thing: Ownership could change from root-to-foo-and-back-again in between Check and Use. We might be talking degenerate cases, again.

... but again^2, probably a huge improvement over ignoring the problem. It will ultimately need OS-API support to avoid these types of issues fully.

As an aside: C++'s filesystem API is (very theoretically) largely unusable due to issues like these. Effectively, almost all of it is UB if even a single other process is doing writes on the filesystem you're accessing.

Why do the files have bad permissions to start with?

Docker is running as root, so the files written in mounted volumes get mapped to uid 0 on the host. When the agent then goes to re-use the checked out code, it can’t run ‘git clean’.

Username space remapping wasn’t adequate, for reasons I’m a bit blurry on. I think recent kernels have some better options on remapping permissions across file systems.

There would traditionally been another TOCTOU is the described solution, namely hardlinks. This can often be used to get root to do something to a file it shouldn't.

The trad solution is to have user writeable areas (home, vartmp, tmp) on different volumes. Some tools have options to not traverse symlinks across volumes for this and other reasons. But on modern systems you are protected by the fs.protected_hardlinks setting.

https://wiki.alpinelinux.org/wiki/Sysctl.conf

Can a user make a hard link to a file they don't own? How does this attack work?

These links (pun unfortunate) should help:

https://sysctl-explorer.net/fs/protected_hardlinks/

http://michael.orlitzky.com/articles/posix_hardlink_heartach...

Answer: fs.protected_hardlinks=1 is what prevents the creation of hardlinks to files you don't own. It's on by default on all machines I checked though.

https://github.com/torvalds/linux/commit/800179c9b8a1e796e44...

Without this, a whole lot of attacks are possible with hardlinks.

Researching this, found [1]:

> POSIX guarantees that hard links can exist. It follows that, on POSIX systems without any non-standard protections, it's unsafe for anyone (but in particular, root) to do anything sensitive in a directory that is writable by another user. Cross-platform programs designed to do so are simply flawed.

I feel like something had to have been lost in translation from Unix to POSIX here. In the original implementation, were users able to hardlink (well, "link") files owned by others? If so, this is a foundational flaw that should probably be fixed across the board with something similar to Linux's non-standard sysctl option.

[1]: http://michael.orlitzky.com/articles/posix_hardlink_heartach...

A link is just a directory entry. If you can write to the directory, you can create entries in it. And back in olden times, mkdir was setuid because it wasn't a syscall.

Yes, normally you could hard link any file if you can read a directory that references it, and if there was a directory you had write access to on the same volume. The hard link doesn't change the permissions of the file.

Hard links are particularly useful for having multiple copies of a directory tree without consuming space for the files. Eg to create an overlay of a build tree, with your changes on top. Unfortunately CoW file systems still have many issues that make this difficult.

why is it running as root anyway?

should probably setuid to the correct user and do the thing there instead

Well in some cases avoiding root might help. But you can have flaws like this root or not, for example Apache httpd still has a known TOCTOU vulnerability with symlinks with a broken check (SymlinksIfOwnerMatch does not actually work).

test -L checks if a file is a symlink; no need for realpath comparisons (which is slower)

> checks if a file is a symlink

to expand on that:

"In computing, a symbolic link (also symlink or soft link) is a file whose purpose is to point to a file or directory (called the "target") by specifying a path thereto."

Yeah but what do you do if your are not using shell?

Hint: it's stat(2) (or equivalent in your language library)

Additional hint: if using pythons os.stat set follow_symlinks to false. It recently took me an embarrassing long time to figure out why my script was failing to find symlinks.

but they were using a shell script in this blog though (and, yes, stat(2) is the way to go there)

Reply on news.ycombinator.com