Rsync.net Technical Notes – Q4 2021
rsync.net
rsync.net
The state of userspace sandboxing across all operating systems I'm aware of is dismal. As a developer it should be trivial to spawn a process with a subset of the privileges of the parent process.
Linux is better than most in terms of options, but you have to understand seccomp, namespaces, cgroups, containers, capabilities, etc etc. It's too confusing. Why can't I just say "spawn this process, limit it to reading files in this directory, writing files in this directory, this socket, max 10GB disk space, max 5% CPU, max 10% memory"? I'd even settle for just being able to limit filesystem and network access.
Even if you do manage to make it work, since the APIs require SETUID executables you might actually be opening yourself up to a bigger risk.
Someone needs to pull a WireGuard and make a small, simple API for userspace sandboxing. And please somehow miraculously make it so it is likely to be implemented by Windows.
Why not you?
It's a breeze on OpenBSD, at least as an application developer. The key is realizing that you can't effectively sandbox a process without the process cooperating. That often means refactoring the source code to reorganize how and when it acquires resources. If you're refactoring the source code, then in many cases it's easier and most effective for the process to sandbox itself, thus pledge(2) (https://man.openbsd.org/pledge.2) and unveil(2) (https://man.openbsd.org/unveil.2). Afterall, the process and its authors are the final and best arbiters of the resources it will need to initialize and acquire during runtime.
All the pain on Linux comes from people struggling to sandbox uncooperative processes (i.e. processes you can't modify and refactor) by trying to construct a fake environment. That's ultimately a losing battle. And quite ironic that people instinctively seek that route even in the context of open source applications where source code is readily available and upstreaming patches and improvements extremely common. OpenBSD has managed to sandbox to some extent almost every program, including command-line utilities, that ship with the OS. And they continue to incrementally improve the those sandboxes over time--IOW, pledge and unveil are amenable to iterative, steady development of capability management, rather than requiring herculean effort up front, potentially resulting brittle architectures that make feature additions excessively difficult.
While there's nothing quite as simple as pledge on Linux (seccomp is close but has several problematic caveats), something very similar to unveil was upstreamed into Linux last year: Landlock. See https://docs.kernel.org/security/landlock.html Unfortunately, AFAICT, Landlock, like seccomp, is mandatorily inherited on fork (no option to disable), which means it can be difficult to integrate into existing applications that might fork or exec other processes--they might be completely unrelated and require a privilege set that isn't a subset of the invoker. Reordering the sequence of process spawning to before the privilege drop can be impractical--effectively impossible if it means basically rewriting the entire program, for example with a read-eval loop (e.g. procmailrc). Spawning processes indirectly through, e.g., systemd or pkgexec, simply shifts the problem and can even increase exposure and bug risk. But for daemon services which are only leaves on the process tree, that's not a concern; Landlock and to a lesser extent seccomp can better approximate unveil and pledge, respectively.
I'm totally fine changing the way I write my applications to make them more secure. I just don't feel like the APIs are there.
Personally I think capability-based security models is the best idea I've seen so far. Set up the file descriptors and network sockets your program needs at startup, then drop any other privileges. The problem is file descriptors don't cover things like directory trees AFAIK.
If performance isn't critical, what are your thoughts on using QEMU to wrap an application for security purposes? That might even work on Windows.
Thanks again to Allan Jude of Klara Systems and Luke Young of bored.engineer for their very interesting comments and explanations.
(these tech notes were published in March of 2022 - they are backward looking to the events of Q4 2021)
Otherwise interesting write-up and pruning unused features I do think helps with security generally.
Can you email ? I actually searched for our conversation for a while and could not find it ... bottom line is that 'mkfifo' is in place and I can't wait to craft up some fascinating remote ssh examples with it ...
For everyone else, the mkfifo command lets you transfer arbitrary streams of data between two systems while using rsync.net as an intermediary.
It makes me happy that it works.
It was, indeed, a very limited test promotion to some small subset of customers and I am still undecided on whether it makes sense to pursue or not.
Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused - specifically by cloud providers. Was it Joyent that dishonored the strongspace lifetime offers ?
I forget ...
Back when Sirius Satellite Radio started, it offered "lifetime" subscriptions. People eventually learned that Sirius meant the lifetime of the radio as long as it tied to the purchaser's account, not the human being.
Many people got Sirius with their new cars. When they started trading them in three or four years later, they learned that their "lifetime" subscription not only didn't go with them to their new car, but the buyer of their old car also didn't get the old subscription.
Today, it wouldn't be as big a deal because you can stream Sirius ($10/month), or some other radio service over the internet. But this was back before even RealPlayer, and the mass adoption of cell phones.
FWIW, I got my first Sirius radio in 1998.
“The first confirmed music transmission from satellite to radio receiver for Sirius was September 1, 2000 in NYC at the current HQ of SiriusXM.”
So what was on in 1998? A precursor to the Sirius service?
This is why we’re hesitant - the bad behavior in this area of pricing and promotion makes any offer like this seem scammy …
Sadly, these things never seem to end well.
I had a few lifetime plans with them. One was with TextDrive, before the Joyent acquisition/merger and that was their shared hosting plan back when they were doing everything on FreeBSD. Then after the merger they did a sort of pre-VPS type plan where it was using Solaris Containers. Those were really cool little boxes.
I also had one of their Strongspace lifetime plans, and that still exists as Expandrive bought them and continues offering the lifetime plans.
Edit: oh, and for how it was ended, Joyent was purchased by Samsung (I think it was) and to get the lifetime stuff off the books they basically forced everyone onto their VPS-like solution but only for ~2 years I think it was. So everything disappeared.
None of those other companies are rsync.net, so it's not really fair of me to consider them when making this decision -- but it made me think it's safer to just pay you guys regularly for a good service. Worst case, I spend a little more over the life of my account, but that money is going to a company I really like, so it's not much of a downside.
> For instance, if you have a pool named TANK and a filesystem named fs1 and you want to store all files smaller than, or equal to, 8K in the special vdev, you would run: ...
This sounds like the files themselves must be of size 8K or less in order to be allocated on the metadata vdev. But doesn't that include all individual file blocks of size 8K or less, not just entire files?
Is it possible to make your service highly available? Seems that is table stakes for other cloud storage services today.
I see several with 80-100 day uptimes right now …
We are announcing an extended, off-peak maintenance window for your backup account.
Beginning Saturday March 5th at 9:00a PT (UTC -8) the server will go offline for approx. 6 hours as we perform data integrity checks and rebalancing.
...That is not a regular, every Saturday maintenance ... perhaps the word "Beginning" made that confusing ?
Let me check with engineering folks and see if they plan on repeating it - but even if they do, it is not a regular maintenance and you should expect your account to have long, uninterrupted uptimes ...