Server-side sandboxing: Containers and seccomp
figma.com
figma.com
(Our python nsjail config for instance: https://github.com/windmill-labs/windmill/blob/main/backend/...)
Irrelevant nit: .proto files are protobuf definition files (like this one: https://github.com/google/nsjail/blob/master/config.proto), a text representation of a specific protobuf contents is typically called (as per man clang-format): .textpb .pb.txt or .textproto - I use .config for examples distributed with nsjail, but it's licentia poetica :)
https://protobuf.dev/reference/protobuf/textformat-spec/#tex...
However, I presume there are now JSON extensions that provide similar functionality?
[1] https://www.freedesktop.org/software/systemd/man/latest/syst...
Can someone with experience with both comment on how (the Linux port of) pledge compares to seccomp? Can it be considered a replacement at this point?
It seems like it could handle the last scenario described in the article fine, since it allows setting granular rwcx permissions on individual paths.
did you consider logging the syscalls invoked during normal usage with 'strace --output=/some/dir -f ...'? This + grep + uniq should make it really simple.
Seccomp and seccomp-bpf are indeed entirely way too limiting. It wasn't really designed for end app developers who are, imo, the ones that should be dictating the policy. The whole lack of pointer deref'ing makes it really difficult for application level developers to make policies that are easier to create.
The promises arg in pledge, https://man.openbsd.org/pledge.2 , does a decent job of grouping related calls together but I think there is a ton of room to make all of this a lot better than it is today.
It also works well if the software developers document what syscalls they rely on and what permissions they need.
When it comes to retrofitting something like pledge (or seccomp) into an existing application when you've not developed it and/or can't easily tell what syscalls are being called then it's always a nightmare.
It doesn't really matter if it's pledge or seccomp at that point (although undoubtedly seccomp is far harder to make use of), if you're doing this kind of security by retroactive whitelist, you're going to have trouble making it work. It's going to take time and effort to implement.
Commercial tools have had it for a long time.. even automatic profiling. Either explicitly profile during a test stage, which is best, or profile-on-first-observation.
In the full automatic mode, which is not optimal but is least effort, any operation performed in the first XX minutes/hours/days are considered 'allowed behavior' and anything after that is denied. Then it will either enforce or 'wait-to-enforce' where enforcement mode only turns on if there are no policy violations in the next XX configurable units of time.
2. Just because a program makes a system call doesn't mean it should. Or should at that moment. A lot of late initialization can be done earlier for tighter policies. Auto traced policies tend to be extremely broad, permitting too much stuff.
Quite the contrary. If the software in question has been written in a remotely sane way, adding some basic pledge restrictions is a matter of adding one line: pledge("stdio rpath whatever you need", NULL) - it usually goes somewhere in main, after setup() but before while(!quit).
You can usually figure out the permission set within a few attempts, even without a very good understanding of the internals, as most (sane) programs will do only a couple of things: an httpd needs to accept connections, read static files, write logs, etc; a window manager needs to talk to X11, open font files, etc; of course there are also complex beasts like Chrome but that one has been done as well.
The *real* challenge is breaking up a complex program (e.g. a streaming music player) into separate processes that are concerned with just one or two things, e.g. separate process to make requests over the network, a separate one to decode media, another to maintain an on-disk cache, and so on. Placing restrictions on these subprocesses is the easy part; figuring out where to draw these lines is what's hard.
This reminds me of a situation where I tried to use firejail to isolate this proprietary piece of software, I ran firejail in the "auto-generate-something-sensible" mode and then tried running it in that profile. It would just randomly break at that point. I never quite figured this out due to lack of time. I was expecting to be able to roll with an auto-generated profile at first and tighten it later, the actual end result was there was no profile at all.
The other issue you run into is getting things which work sometimes and then stop working randomly. Especially when it's a large graphical application. It will do something strange when you click a specific button and crash. Now you are annoyed, probably not in the mood to debug this, so maybe you make a note for later. Now you have to recreate the issue under strace, figure out what you need to pledge now, and repeat.
Yes, if you're trying to pledge ls, it's pretty easy. If you're trying to pledge anything non-trivial (i.e. anything which would _really_ benefit from these security restrictions) you end up iterating a lot.
It's not very fun.
This is exactly the point where the experience between pledge and e.g. firejail drastically diverges. The entire reason why pledge is so nice, is because it assumes source access. You can use execpromises to "jail" something you don't control, but the set of promises is always going to be unnecessarily broad, as even the sanest software out there often needs a tiny backflip before it can enter the main loop. Source access also means you have the means to investigate what exactly went wrong, or to actually fix the stupidity (rather than broadening the privileges).
The amount of things you can do to a proprietary blob to contain it is pretty limited - by definition! I think using a container/VM to completely isolate it would be a better call.
> It will do something strange when you click a specific button and crash. Now you are annoyed, probably not in the mood to debug this, so maybe you make a note for later.
When you do pledge("... error", ...), the app will get errors from disallowed syscalls, rather than a SIGABRT, which is useful when you're not sure. Mis-handling an error can still blow things up, but that's a sign that maybe the overall code quality is not great. In any case, yes you do basically need to be running the application under ktrace (/strace) for as long as you're not certain.
Actually what I think would be great is a "tracing mode" for pledge, where the kernel reports violations (PID + syscall with parms + suggested promise), maybe even takes a core snapshot at each violation, but otherwise doesn't hurt the application.
> If you're trying to pledge anything non-trivial (i.e. anything which would _really_ benefit from these security restrictions) you end up iterating a lot.
Indeed, but that's a curse of complex software, not a shortcoming of pledge itself. OpenBSD introduced pledge and contained almost the entire base system within a single release cycle.
I personally think that as an industry, software is going through a crisis of explosive complexity. I see efforts like pledge as a mirror, through which that complexity stares back at us, in all of its ugliness. Blaming the mirror does not address the issue.
I was never trying to imply that pledge had shortcomings.
I don't think we're disagreeing on anything here anyway.
The point I was making is that if you've got a large and complicated piece of software, which you didn't write yourself, which wasn't written with the intention of someone implementing a syscall filter for it, you will have a bad time. It's not quite as bad as if you have the code but it's always going to be pretty bad regardless.
I think pledge is great, and the rollout was really good (I use OpenBSD for my home router and for some other infrastructure). The OpenBSD developers were in the beneficial position that they are already familiar with the source code for their base userland, they already regularly audit and maintain security improvements for it. Also noteworthy is the fact that most of the OpenBSD base is (intentionally) not formed of extremely complex software.
It works well with openbsd because its standard, and most if not all openbsd packages make use of it
Though maybe I'm misunderstanding how Linux pledge works. I'm only familiar with the openbsd usage of it
[1] https://github.com/google/nsjail [2] https://github.com/containers/bubblewrap
It seems like something is broken, and we are all patching things up piecemeal.
Funny enough, we've gone the reverse path for LLM AI-generated code sandboxing for louie.ai / Graphistry . We started with container isolation with careful network, volume, compute etc enablement first, and only now adding nsjail to the runners within the container as an extra defense layer.
The negative space is interesting too. We initially explored alternatives like wasm (too slow and underpowered for our generated python GPU analytics workloads) and firecracker vm (too unwieldy and unportable for our small team). As we do more k8s and enable more interactive data viz customization + web-scale static serving, would love to revisit both.
On which note, we have a bit of budget for someone to help harden the nsjail layer, if of interest!
Liquid Metal from Weaveworks seems interesting but I don't even know where I would start.
I believe part is portability, as they may require nested virtualization features to be available, and maybe QEMU overhead. Maybe also something about use in China vs elsewhere?
They (and QEMU) have been around a long time...
I believe the next generation looks something like landlock (https://docs.kernel.org/userspace-api/landlock.html).