Toro: Deploy Applications as Unikernels
github.com
github.com
Docker was conceived to solve the problem of things "working on my machine", and not anywhere else. This was generally caused by the differences in the configuration and versions of dependencies. Its approach was simple: bundle both of these together with the application in unified images, and deploy these images as atomic units.
Somewhere along the lines however, the problem has mutated into "works on my container host". How is that possible? Turns out that with larger modular applications, the configuration and dependencies naturally demand separation. This results in them moving up a layer, in this case creating a network of inter-dependent containers that you now have to put together for the whole thing to start... and we're back to square one, with way more bloat in between.
Now hardware virtualization. I like how AArch64 generalizes this: there are 4 levels of privilege baked into the architecture. Each has control over the lower and can call up the one immediately above to request a service. Simple. Let's narrow our focus to the lowest three: EL0 (classically the user space), EL1 (the kernel), EL2 (the hypervisor). EL0, in most operating systems, isn't capable of doing much on its own; its sole purpose is to do raw computation and request I/O from EL1. EL1, on the other hand, has the powers to directly talk to the hardware.
Everyone is happy, until the complexity of EL1 grows out of control and becomes a huge attack surface, difficult to secure and easy to exploit from EL0. Not good. The naive solution? Go a level above, and create a layer that will constrain EL1, or actually, run multiple, per-application EL1s, and punch some holes through for them to still be able to do the job—create a hypervisor. But then, as those vaguely defined "holes", also called system calls and hyper calls, grow, won't so the attack surface?
Or in other words, with the user space shifting to EL1, will our hypervisor become the operating system, just like docker-compose became a dynamic linker?
I think you're over-egging the pudding. In reality, you're unlikely to use more than 2 types of container host (local dev and normal deployment maybe), so I think we've moved way beyond square 1. Config is normally very similar, just expressed differently, and being able to encapsulate dependencies removes a ton of headaches.
Unfortunately containers have always had an absolutely horrendous security story and they degrade performance by quite a lot.
The hypervisor is not going away anytime soon - it is what the entire public cloud is built on.
While you are correct that containers do add more layers - unikernels go the opposite direction and actively remove those layers. Also, imo the "attack surface" is by far the smallest security benefit - other architectural concepts such as the complete lack of an interactive userland is far more beneficial when you consider what an attacker actually wants to do after landing on your box. (eg: run their software)
When you deploy to AWS you have two layers of linux - one that AWS runs and one that you run - but you don't really need that second layer and you can have much faster/safer software without it.
The story is quite different in HP-UX, Aix, Solaris, BSD, Windows, IBM i, z/OS,...
There are AppContainers. Those have existed for a while and are mostly targeted at developers intending to secure their legacy applications.
https://learn.microsoft.com/en-us/windows/win32/secauthz/app...
There's also Docker for Windows, with native Windows container support. This one is new-ish:
https://learn.microsoft.com/en-us/virtualization/windowscont...
Perf is much better on Windows server. It's actually really pleasant to get your office appliances (a build agent etc) in a container on a beefy Windows machine running Windows server.
Doesn’t “virtualization-based security” mean everything does, container or no? Or are they actually VMs even with VBS disabled?
AppContainers, and Docker for Windows (the one for running dockerized windows apps, not running linux docker containers on top of WSL) is using this API, these high-level features are just the 'porcelain'
Very useful if you are packaging trusted software don't want to upgrade your windows server license.
Aren't there ways of overwriting the existing kernel memory/extending it to contain an a new application if an attacker is able to attack the running unikernel?
What protections are provided by the unikernel to prevent this?
What becomes harder is if you have a binary that forces you to rewrite the program in memory as you suggest. That's where classic page protections come into play such as not exec'ing rodata, not writing to txt, not exec'ing heap/stack, etc. Just to note that not all unikernel projects have this and even if they do it might be trivial to turn them off. The kernel I'm involved with (Nanos) has other features such as 'exec protection' which prevents that app from exec-mapping anything not already explicitly mapped exec.
Running arbitrary programs, which is what a lot of exploit payloads try to achieve, is pretty different than having to stuff whatever they want to run inside the payload itself. For example if you look at most malware it's not just one program that gets ran - it's like 30. Droppers exist solely to load third party programs on compromised systems.
Does this mean JIT (and I guess most binary instrumentation (debuggers) / virtualization / translation tech) won't run as expected?
Suppose you control the entire stack though, from the bare metal up. (Correct me if I'm wrong, but) Toro doesn't seem to run on real hardware, you have to run it atop QEMU or Firecracker. In that case, what difference does it make if your application makes I/O requests through paravirtualized interfaces of the hypervisor or talks directly to the host via system calls? Both ultimately lead to the host OS servicing the request. There isn't any notable difference between the kernel/hypervisor and the user/kernel boundary in modern processors either; most of the time, privilege escalations come from errors in the software running in the privileged modes of the processor.
Technically, in the former case, besides exploiting the application, a hypothetical attacker will also have to exploit a flaw in QEMU to start processes or gain further privileges on the host, but that's just due to a layer of indirection. You can accomplish this without resorting to hardware virtualization. Once in QEMU, the entire assortment of your host's system calls and services is exposed, just as if you ran your code as a regular user space process.
This is the level you want to block exec() and other functionality your application doesn't need at, so that neither QEMU nor your code ran directly can perform anything out of their scope. Adding a layer of indirection while still leaving user/kernel, or unikernel/hypervisor junction points unsupervised will only stop unmotivated attackers looking for low-hanging fruit.
One of the things that might not be so apparent is that when you deploy these to something like AWS all the users/process mgmt/etc. gets shifted up and out of the instance you control and put into the cloud layer - I feel that would be hard to do with physical boxen cause it becomes a slippery slope of having certain operations (such as updates) needing auth for instance.
Hypervisors expose a much smaller API surface area to their tenants than an operating system does to its processes which makes them much easier to secure.
When you say "high security" do you mean Confidential Computing workloads run by Trusty (Enclave) / Virtee (Realm) etc? If so, aren't these system limited in what they can do, as in, there usually is another full-blown OS that's running the user-facing bits?
> that need to protect against state actors
This is a very high bar for a software-only solution (like a microkernel) to meet? In my view, open hardware specification, like OpenTitan, in combination with small-ish software TCB, make it hard for state actors (even if not impossible).
There have been many such systems verified to be secure against state actors according to the TCSEC Orange Book Level A1 standard and the subsequent Common Criteria SKPP standard which requires both full formal proofs of security and explicitly requires the NSA to identify zero vulnerabilities during a multi-month penetration test before allowing usage in NSA and DoD systems.
Some unikernels are intended to run under a hypervisor or on bare metal. Bare metal means you need some drivers, but if you have a use case for a unikernel on bare metal, you probably don't need to support the vast universe of devices, maybe only a few instances of a couple types of things.
I've got a not production ready at all hobby OS that's adjacent to a unikernel; runs in virtio hypervisors and on bare metal, with support for one NIC. In it's intended hypothetical use, it would boot from PXE, with storage on nodes running a traditional OS, so supporting a handful of NICs would probably be sufficient. Modern NICs tend to be fairly similar in interface, so if the manufacturer provides documentation, it shouldn't take too long to add support at least once you've got one driver doing multiple tx/rx queues and all that jazz... plus or minus optimization.
For storage, you can probably get by with two drivers, one for sata/ahci and one for nvme. And likely reuse an existing filesystem.
Do you usually publish your hobby code publicly? If not, consider this an appeal to do so (:
> Modern NICs tend to be fairly similar in interface, so if the manufacturer provides documentation, it shouldn't take too long to add support at least once you've got one driver ... For storage, you can probably get by with two drivers
I take that there aren't any pluggable drivers for NICs like there's for nvme/sata disks?
Yes; https://github.com/russor/crazierl/ and there's an in browser demo as well https://crazierl.org/demo.html (thanks to v86! https://github.com/copy/v86 ) Supports virtio-net and realtek 8168.
> I take that there aren't any pluggable drivers for NICs like there's for nvme/sata disks?
I mean, there is NDIS / NDISWrapper. Or, I think it wouldn't be too hard to run netbsd drivers... but I'm crazy and want my drivers in userland, in Erlang, so none of that applies. :)
As a fair warning, there's some concurrency errors in the kernel which I haven't tracked down that results in sometimes getting stuck before the shell prompt comes up, the tcp stack is just ok enough to mostly work, and the dhcp client only works if everything goes right.
I've been dabbling in Erlang and OS development myself, my biggest inspirations being Microsoft Singularity and QNX. The former is a C# lookalike of what you're making, or at least that's how it seems from my perspective.
The readme mentions a FreeBSD-like system call interface, but then the drivers and the network stack are written in Erlang, and, as you've mentioned, run in the user land. Is that actually a unikernel design with BEAM running in the kernel, or more of a microkernel hosting BEAM, with it providing device handling and the user space?
Where that fits in the taxonomy of life, I'm not sure. There is a kernel/userspace boundary (and also a c-code/erlang code boundary in userspace), so it's not quite a unikernel. I wouldn't really call it a microkernel; there's none of the usual microkernel stuff... I just let userspace do i/o ports with the x86 task structure and do memory mapped i/o by letting it mmap anything (more or less). The kernel manages timers/timekeeping and interrupts, Erlang drivers open a socket to get notified when an interrupt fires --- level triggered interrupts would be an issue. Kernel also does thread spawning and mutex support, connects pipes, early/late console output, etc.
If I get through my roadmap (networked demo, uefi/amd64 support, maybe arm64 support, run the otp test suite), I might look again and see if I can eliminate the kernel/userspace divide now that I understand the underneath, but the minimal kernel approach lets me play around with the fun parts, so I'm pretty happy with it. I've got a slightly tweaked dist working and can hotload userspace Erlang code over the network, including the tcp stack, which was the itch I wanted to scratch... nevermind that the tcp stack isn't very good at the moment ;)
>I just let userspace do i/o ports [...] and do memory mapped i/o by letting it mmap anything (more or less). The kernel manages timers/timekeeping and interrupts [...]
This is how QNX does it too, allowing privileged processes to use MAP_PHYS and port I/O instructions on x86, and handle interrupts like they're POSIX signals. It all boils down to how you structure your design, but personally, I think that's not a bad approach at all. The cool thing about it is that, after the initial setup, you can drop the privileges for creating further mappings and handlers, reducing the attack surface.
Unless you're trying to absolutely minimize the cost and amount of context switches, I think moving BEAM into the kernel would be a downgrade, but again, I'm a big proponent of microkernels :)
Looking forward to the UEFI and AArch64 ports!
Toro can run on baremetal although I stopped to support on that a few years ago. I tagged in master the commit when this happened. Also, I removed the TCP/IP Stack in favor to VSOCK. Those changes, though, could be reversed in case there is interest on those features.
Python package management is a disaster. There should be ways of having multiple versions of a package coexist in /usr/lib/python, nicely organized by package name and version number, and import the exact version your script wants, without containerizing everything.
Electron applications are the other type of "fuck it" solution. There should be ways of writing good-looking native apps in JavaScript without actually embedding a full browser. JavaScript is actually a nice language to write front-ends in.
Sometimes, the reduction of development friction is the only reason a product ends up in your hands.
I say this as someone whose professional toolkit includes Docker, Python and Electron; Not necessarily tools of choice, but I'm one guy trying to build a lot of things and life is short. This is not a free lunch and the optimizer within me screams out whenever performance is left on the table, but everything is a tradeoff. And I'm always looking for better tools, and keep my eyes on projects such as Tauri.
Curious, can you expand on this?
To be fair, the UNIX wars probably inspired this in the first place - outside of SVR4 deriviatives, most commercial UNIX systems (HP-UX, AIX, Tru64) had their own packaging format. Even the gratis BSD systems all have their own variants of the same packaging system. This was the one thing that AT&T and Sun Solaris got right. Linux distros merely followed suit at the time - Redhat with RPM, Debian with DEB, and then Slackware and half a dozen other systems - thankfully we seem to have coalesced on RPM, DEB, Flatpak, Snap, Appimage etc... but yeah that's before you get to the language specific package management. It's a right mess, carried over from 90's UNIX "NIH" syndrome.
They can just say "here's the source code, here's a container where it works, the rest is the OS maintainer's job, and if Debian users running 10 year old software bug me I'm just gonna tell them to use the container"
Have you tried uv?
import torch==2.9.1
I've written my fair share of GUIs, and React (and thus Javascript) is great compared to, I don't know, PHP, but CSS is the absolute devil.
This is demonstratably untrue.
In terms of the security aspects though, how does security holes in a layer that restricts things more than without it degrade security? Seems like saying that CVEs on browser's javascript sandboxing degrade the browser security more than just not having sandboxes.
The problem with "container" security is that even in this thread many people seem to think that it is a security barrier of some kind when it was never designed to be one. The v8 sandbox was specifically created to deal with sandboxing. It still has issues but at least it was thought about and a lot of engineering went into it. Container runtimes are not exported via the kernel. Unshare is not named 'create_container'. A lot of the container issues we see are runtime issues. There are over a half-dozen different namespaces that are used in different manners that expose hard to understand gotchas. The various container runtimes decide themselves how to deal with these and they have to deal with all the issues in their code when using them. A very common bug that these runtimes get hit by are TOCTOU (time of check to time of use) vulns that get exposed in these runtimes.
Right now there is a conversation about the upcoming change to systemd that runs sshd on vsock by default (you literally have to disable it via kernel cli flag - systemd.ssh_auto=no) - guess what one of the concerns is? Vsock isn't bound to a network namespace. This is not itself a vulnerability but it most definitely is going to get taken advantage in the future.
What does that have to do with unikernel vs more traditional VMs? You can build a rootfs that doesn't have any interactive userland. Lots of container images do that already.
I am not a security researcher, but I wouldn't think it would be too hard to load your own shell into memory once you get access to it. At least, compared to pulling off an exploit in the first place.
I would think that merging kernel and user address spaces in a unikernel would, if anything, make it more vulnerable than a design using similar kernel options that did not attempt to merge everything into the kernel. Since now every application exploit is a kernel exploit.
Also merging the address space is not a necessity. In fact - 64-bit (which is essentially all modern cloud software) mandates virtual memory to begin with and many unikernel projects support elf loading.
I've achieved near-native performance for intensive activities like gaming, music and visual production. Hardware acceleration is kind of a mess but using tricks like GPU passthrough for multiple cards, dedicated audio cards and and block device passthrough, I can achieve great latency and performance.
One benefit of this is that my desktop acts as a mainframe, and streaming machines to thin clients is easy.
My model for a long time has been not to trust anything I run, and this allows me to keep both my own and my client's work reasonably safe from a drive-by NPM install or something of that caliber.
Now that I also use a Apple Silicon MacBook as a daily driver, I very much miss the comfort of a fully virtualized system. I do stream in virtual machines from my mainframe. But the way Tahoe is shaping up, I might soon put Asahi on this machine and go back to a fully virtualized system.
I think this is the ideal way to do things, however, it will need to operate mostly transparently to an end user or they will quickly get security fatigue; the sacrifices involved today are not for those who lack patience.
Also, relevant XKCDs:
https://www.explainxkcd.com/wiki/index.php/2044:_Sandboxing_...
Are you for real? Tell us you've never worked on a mainframe without telling us you've ever worked on a mainframe.
* hot-swap power & CPUs
* RAS (Reliability, Availability, Serviceability)
* vendor SLAs
* fault containment
* designed uptime vs achieved uptime
mainframe (noun)
main· frame ˈmān-ˌfrām
1: a large, powerful computer that can handle many tasks concurrently and is usually used commercially
2: (dated): a computer with its cabinet and internal circuits especially when considered separately from any peripherals connected to the computer
The features you list are great to have, but my setup fits the first definition of mainframe as described. If you feel this definition is not specific enough, email Merriam-Webster and don't bother me about it.
What worries me is when this model escalates from being cobbled up together by a system administrator with limited resources, to becoming baked into the design of software; the appropriation of the hypervisor layer by software developers who are reluctant to untangle the mess they've created at the user/kernel boundary of their program and instead start building on top of hardware virtualization for "security", to ultimately go on and pollute the hypervisor as the level of host OS access proves insufficient. This is beautifully portrayed by the first XKCD you've linked. I don't want to lose the ability to securely run VMs as the interface between the host and the guest OSes grows just as unmanageable as that of Linux and BSD system calls and new software starts demanding that I let it use the entirety of it, just like some already insists that I let it run as root because privilege dropping was never implemented.
If you develop software, you should know what kind of operating system access it needs to function and sandbox it appropriately, using the operating system's sandboxing facilities, not the tools reserved for system administrators.
At some point, few people even understand the whole system and whether all these layers are actually accomplishing anything.
It’s especially bad when the code running at rarified levels is developed by junior engineers and “sold” as an opaque closed source thing. It starts to actually weaken security in some ways but nobody is willing to talk about that.
“It has electrolytes…”
From what I read, I gather nixpkgs are more hermetic (as in Bazel [0]) & not reproducible? https://discourse.nixos.org/t/nixos-is-not-reproducible/4268... / https://archive.vn/mXeih
Yea, with uneeded bload like rule based access controls, ACS and secret management. Some comments on this site.
The difference is that you can move that whole bunch of interlinked containers to another machine and it will work. You don't get that when running on bare hardware. The technology of "containers" is ultimately about having the kernel expose a cleaned up "namespaced" interface to userspace running inside the container, that abstracts away the details of the original machine. This is very much not intended as "sandboxing" in a security sense, but for most other system administration purposes it gets pretty darn close.
Containers arose as a way to solve the dependency problems created by traditional Unix. They grow from tools like chroot, BSD jails, and Solaris Zones. Containers allow to deploy dependencies that cannot be simultaneously installed on a traditional Unix host system. it's not a UNIX architecture limitation but rather a result of POSIX + tradition; e.g. Nix also solves this, but differently.
Containers (like chroot and jail before them) also help ensure that a running service does not depend on the parts of the filesystem it wasn't given access to. Additionally, containers can limit network access, and process tree access.
These limitations are not a proper security boundary, but definitely a dependency boundary, helping avoid spaghetti-style dependencies, and surprises like "we never realized that our ${X} depends on ${Y}".
Then, there's the Fundamental Theorem of Software Engineering [1], which states: "We can solve any problem by introducing an extra level of indirection." So yes, expect the number of levels of indirection to grow everywhere in the stack. A wise engineer can expect to merge or remove a some levels here and there, when the need for them is gone, but they would never expect that new levels of indirection should stop emerging.
[1]: https://en.wikipedia.org/wiki/Fundamental_theorem_of_softwar...
I've mostly focused on the worst Docker horrors I've seen in production, extrapolating that to the future of containers, as pulling in new "containerized" dependencies will inevitably become just as effortless as it currently is with regular dependencies in the new-style high-level programming languages. You've primarily described a relatively fresh, or a well-managed Docker deployment, while admitting that spaghetti-style dependencies have become a norm and new layers will pile up (and by extension, make things hard to manage).
I think our points of view don't actually collide.
Many of the "new" languages, like Go (16 years), Rust (13 years), or Zig (9 years) just can build static binaries, not even depending on libc. This has both upsides and downsides, especially with security fixes. Rebuilding a container to include an updated .so dependency is often easier and faster than rebuilding a Rust project.
Docker (or preferably Podman) is not a replacement for linkers. It's an augmentation to the package system, and a replacement for the common file system layout, which is inadequate for modern multi-purpose use of a Unix (well, Linux) box.
As for the linker analogy, I compared docker-compose (not Docker proper) to a dynamic linker because it's often used to bring up larger multi-container applications, similar to how large monolithic applications with plenty of shared library dependencies are put together by ld.so, and those multi-container applications can be similarly brittle if developed under the assumption that merely wrapping them up in containers will assure portability, defeating most of Docker's advantages and reducing it to a pile of excess layers of indirection. This is similar to the false belief that running kernel-mode code under a hypervisor is by itself more secure than running it as process on top of a bare-metal kernel.
> Everyone is happy, until the complexity of EL1 grows out of control and becomes a huge attack surface, difficult to secure and easy to exploit from EL0. Not good. The naive solution? Go a level above, and create a layer that will constrain EL1, or actually, run multiple, per-application EL1s, and punch some holes through for them to still be able to do the job—create a hypervisor. But then, as those vaguely defined "holes", also called system calls and hyper calls, grow, won't so the attack surface?
(Disclaimer: this is all very far from my area of expertise, so some of the below may be wrong/misleading)
Nobody can agree whether microkernels or monolithic kernel are "better" in general, but most people seem to agree that microkernels are better for security [0], with seL4 [1] being a fairly strong example. But microkernels are quite a bit slower, so in the past when computers were slower, microkernels were noticeably slower, and security was less of a concern than it is now, so essentially every mainstream operating system in the 90s used some sort of monolithic kernel. These days, people might prefer different security–performance tradeoffs, but we're still using kernels designed in the 90s, so it isn't easy to change this any more.
Moving things to the hypervisor level lets us gain most of the security benefits of microkernels while maintaining near-perfect compatibility with the classic Linux/NT kernels. And the combination of faster computers (performance overheads therefore being less of an issue), more academic research, and high-quality practical implementations [2] means that I don't expect the current microkernel-style hypervisors to gain much new attack surface.
This idea isn't without precedent either—Multics (from the early 70s) was partially designed around security, and used a similar design with hardware-enforced hierarchical security levels [3]. Classic x86 also supports 4 different "protection rings" [4], and virtualisation plus nested virtualisation adds 2 more, but nothing ever used rings 1 and 2, so adding virtualisation just brings us back to the same number of effective rings as the original design.
[0]: https://en.wikipedia.org/wiki/Microkernel#Security
[2]: https://xenproject.org/projects/hypervisor/
The idea that microkernels are slow came from analyzing a popular microkernel at the time - mach. It in no way is a true blanket statement for all microkernels.
edit: found a good comparison chart (first link)
https://sigops.org/s/conferences/sosp/2013/talks/elphinstone...
https://sel4.systems/About/seL4-whitepaper.pdf
https://sel4.systems/performance.html
Don't microkernels inherently require lots of context switches between kernel-space and user-space, which are especially slow in a post-Meltdown/Spectre world? I know that Linux has semi-recently added kTLS and KSMBD to speed up TLS/SMB, and Windows used to implement parts of font rendering and its HTTP server in kernel mode to speed things up too, so this gave me the impression that having more things inside the kernel (== more monolithic) is better for speed. Or is this only the case because of how the Linux/NT kernels are implemented, and doesn't apply to microkernels?
I do think there is very interesting research in this topic however, on how systems could mitigate some of these attacks in a more foolproof way.
https://microkerneldude.org/2024/04/18/gofetch-will-people-e...
https://trustworthy.systems/publications/abstracts/Wistoff_S...
[0]: https://www.tritondatacenter.com/blog/unikernels-are-unfit-f...
Unikernels don't work for him; there are many of us who are very thankful for them.
Why? Can you explain, in light of the article, and for those of us who may not be familiar with qubes-mirage-firewall, why?
Networking also happens in its own VM, and you can have multiple VMs dedicated to networking.
Much lower memory footprint running mirage firewall, and an attack surface orders of magnitude smaller (compared to a VM running a Linux distribution purely for networking).
I think unikernels potentially have their place, but as he points, they remain mostly untried, so that's fair. We should analyze why that is.
On performance: I think the kernel makes many general assumptions that some specialized domains may want to short circuit entirely. In particular I am thinking how there's a whole body of research of database buffer pool management basically having elaborate work arounds for kernel virtual memory subsystme page management techniques, and I suspect there's wins there in unikernel world. Same likely goes for inference engines for LLMs.
The Linux kernel is a general purpose utility optimizing for the entire range of "normal things" people do with their Linux machines. It naturally has to make compromises that might impact individual domains.
That and startup times, big world of difference.
Is it going to help people better sling silly old web pages and whatever it is people do with computers conventionally? Yeah, I'd expect not.
On security, I don't think it's unreasonable or pure "security theatre" to go removing an attack surface entirely by simply not having it if you don't need it (no users, no passwords, no filesystem, whatever). I feel like he was a bit dismissive here? That is also the principle behind capability-passing security to some degree.
I would hate to see people close the door on a whole world of potentials based on this kind of summary dismissal. I think people should be encouraged to explore this domain, at least in terms of research.
Why? The solution thus far has been to turn off what the kernel does, and, do those things in userspace, not move everything in the kernel? Where are these performance gains to be had?
> The Linux kernel is a general purpose utility optimizing for the entire range of "normal things" people do with their Linux machines.
Yeah, like logging and debugging. Perhaps you say: "Oh we just add that logging and debugging to the blob we run". Well isn't that now another thing that can take down the system, when before it was a separate process?
> That and startup times, big world of difference.
Perhaps in this very narrow instance, this is useful, but what is it useful for? Can't Linux or another OS be optimized for this use case without having to throw the baby out with the bathwater? Can't one snapshot a Firecracker VM and reach even faster startup times?
> On security, I don't think it's unreasonable or pure "security theatre" to go removing an attack surface entirely
Isn't perhaps the most serious problem removing any and all protection domains? Like between apps and the kernel and between the apps themselves?
I mean -- sure maybe remove the filesystem, but isn't no memory protection what makes it a unikernel? And, even then, a filesystem is usually a useful abstraction! When have I found myself wanting less filesystem? Usually, I want more -- like ZFS.
This is all just to say -- you're right -- there may be a use case for such systems, but no one has really adequately described what that actually is, and therefore this feels like systems autoeroticism.
There's all sorts of jankin' about trying to squeeze ounces of performance out of the kernel's page management, specifically for buffer pools.
e.g. https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...
Page management isn't really a thing we can do well "in user space". And the kernel has strong ideas about how this stuff works, which work very well in the general case. But a DB (or other system level things like garbage collectors, etc) are special cases, often with special needs.
LeanStore, Umbra, etc. do tricks with VMM overcommit and the like to fiddle around with this, and the above paper even proposes custom kernel modules for the purpose (There's a github repo associated, I'd have to go look).
And then, further, a DB goes and basically implements its own equivalent of a filesystem, managing its own storage. Often fighting with the OS about the semantics of fsync/durability, etc.
I don't think it's an unreasonable mental leap for people to start thinking: "I'm by necessity [cuz cloud] in a VM. Now I'm inside an OS in a VM, and the OS is sometimes getting in my way, and I'm doing things to get around the OS... Why?"
But it is the thing most high performance OLTP DBMSs, most of us are aware of, do? I'm also not sure your cite is relevant here. Or it is at least niche. The comparison is made to LeanStore, which is AFAICT is not feature complete, and a research prototype?
Your cite does not describe a unikernel use case, but instead in kernel helper modules. Your cite is about leveraging in kernel virtual memory for the DB buffer cache, and thus one wonders how sophisticated the VM subsystem is in most unikernels? That is -- how is this argument for unikernels? Seems as though your cite is making the opposite argument -- for a more complex relationship between the DB and the kernel, not a pared down one.
> And then, further, a DB goes and basically implements its own equivalent of a filesystem, managing its own storage. Often fighting with the OS about the semantics of fsync/durability, etc.
The fights you're describing what have thus far been the problems of ceding control of the buffer cache to the kernel, via mmap, especially re: transactional safety.
If your argument is kernels may need a bottom up redesign to make ideas like this work, I suppose that makes sense. However, again, I'm not sure that makes unikernels more of an answer here than anywhere else, though.
> I don't think it's an unreasonable mental leap for people to start thinking: "I'm by necessity [cuz cloud] in a VM. Now I'm inside an OS in a VM, and the OS is sometimes getting in my way, and I'm doing things to get around the OS... Why?"
I think that's a fair thought to have, but the problem is how it actually works in practice. As in, less code seems really enticing, the problem is what abstractions are you throwing away. If the abstraction is less memory protection, maybe this is not a good tradeoff in practice.
` If this approach seems fringe, things get much further afield with language-specific unikernels like MirageOS that deeply embed a particular language runtime. On the one hand, allowing implementation only in a type-safe language allows for some of the acute reliability problems of unikernels to be circumvented. On the other hand, hope everything you need is in OCaml! `
ToroKernel is written in freepascal.
All of the text before and after is completely irrelevant
but at the same time i do think it is fair at this juncture to compare the tech to things like wasm, with which unikernels are much more of a direct competitor than containers. it is ironic because i can already hear in my head the hilarious tirade he would unleash about how horrific docker is in every way, debugging especially, but yet somehow this is worse for containers than for unikernels. my view at the present is that unikernels are fantastic for software you trust well enough to compile down to the studs, and the jury is still out on their efficacy beyond that. but holy fuck i seear to god i have spent more time fucking with docker than many professional programmers have spent learning their craft en toto, and i have nothing to show for it. it sucks every time, no matter what. my only gratitude for that experience revolves around (1) saving other peoples’ time on my team (same goes for git, but git is, indisputably, a “good” technology, all things considered, which redeems it entirely), and (2) it motivated me to learn about all the features that linux, systemd, et al. have (chroot jails, namespaces, etc.) in a way that surely exceeds my natural interest level.
Nah not really what he's saying. He's saying that if you throw out all the security affordances provided by page tables and virtual memory, it outweighs the "profound advantage" (which as he mentions, is arguable anyway since user/kernel context switch is a negligible cost in most modern systems).
You're selling a great deal in order to buy not much. It's a poor tradeoff.
I'm surprised to read that unikernels would start up much faster than containers. It seems like a unikernel needs to do more work (load kernel, and load app), in a more restricted way (hypervisor) than simply loading the app in a cgroup + namespace and letting it rip.
Are you sure this is an apples to apples comparison of similarly optimized images?
At a conceptual level I really disagree with this piece, though:
> one cannot play up Linux kernel vulnerabilities as a silent menace while simultaneously dismissing hypervisor vulnerabilities as imaginary.
One can reasonably recognize Linux kernel vulnerabilities as extant and pervasive while acknowledging that hypervisors can be vulnerable. One can also realize that the surface area exposed by Linux is fundamentally much larger than that exposed by most hypervisors, and that the Linux `unshare` mechanism is insecure by default. It's kind of funny - the invocation of Linux really undermines this argument; there's no _reason_ a process / container isolation based system should be completely broken, but Linux _is_, and so it becomes a very weak opponent.
I really don't think I can agree with the debugging argument here at a conceptual level, either. Issues with debugging unikernels are caused by poor outside-in tooling, but with good outside-in tooling, a unikernel should be _easier_ to debug than a container or OS process, because the VM-owner / hypervisor will often already have a way to inspect the unikernel-machine's entire state from the outside, without additional struggle of trying to maintain, juggle, and restore multiple contexts within a running system. There is essentially an ISP/ICE debugging probe attached to the entire system end to end by default, in the form of the hypervisor.
For example, there is no reason a hosting hypervisor could not provide DTrace in a way which is completely transparent to the unikernel guest, and this would be much easier to implement than DTrace self-hosted in a running kernel!
If done properly, this way a uni-application basically becomes debugging-agnostic: it doesn't need cooperative tracepoints or self-modifying patches (and all of the state juggling that comes with that, think like Kprobe), because the hypervisor can do the tracing externally. The unikernel does not need to grow (in security surface area, debug-size, blast radius, etc.) to add more trace and debug capability.
By which I mean I see two variants:
1- exotic and interesting and constrained but probably not applicable for people in the form of e.g. MirageOS. not applicable because OCaml just isn't mainstream enough
2- Or other systems which allow much easier porting of existing systems by providing a libc and extended set of "porting" libraries which end up by recreating huge swathes of what the operating system is doing already anyways, in order to make the existing application just cross compile and "feel at home". But in reality probably always in an incomplete or odd way, and now you're using someone's hand crafted set of compatibility libraries instead of a battle tested operating system.
I just think we haven't seen the right system, yet, which would probably be some specific application development mostly from the ground up in the context of unikernel, not the other way around. Potentially a set of constrained and targeted Rust etc crates built from nostd up + some services. I kept looking for MirageOS for Rust and haven't seen, instead saw stuff more like 2.
You also have to recreate your entire userspace and debugging tools to work in this environment, and testing or even running or debugging your software is also a headache.
Still difficult to see how the unikernel could be slower, but I doubt the difference would be huge? Don't have anything to back that up though.
Also always a little wary of projects that have bad typos or grammar problems in a README - in particular on one of the headings (thought it's possible these are on purpose?). But that's just me :\
Neat.
I was not expecting Pascal, thats an interesting choice. One thing I do like is that Freepascal has one of the better ways of making GUIs meanwhile every other language had decided that just letting Javascript build UIs is the way.
Now I write Javascript and SQL.
:)
As much as I'm nostalgic about Pascal and my childhood... I'd personally prefer OCaml.
Plus these might be smaller and might run faster than containers too.
Faster might be possible without the context switching between kernel and app? And maybe additional opportunities for the compiler to optimize the entire thing (e.g., LTO)?
if you're not sure which you want its probably container
Not really. Separation from (type 1) hypervisor (or rather distrust of the host [0]) requires hardware support; ex: ARM CCA / AMD SEV-SNP / Intel TDX.
For separation from the supervisor, Android developed a peculiar approach in "pKVM" for ARM where the host (supervisor) is partitioned away from the guest [1].
Both those "separations" is not something Toro provides on its own; the Toro unikernel would totally be under the control of the host, from what I can tell. That said, what Toro (or any unikernel, really) does is reduce the attack surface area, as the (guest) supervisor is pruned to run just one particular application (more code to partition things up will eliminate a class of attacks but may result in new attack vectors [2]).
[0] ex: https://news.ycombinator.com/item?id=44678249
[1] Protected KVM on Arm64: A Technical Deep Dive - Quentin Perret, Google https://www.youtube.com/watch?v=9npebeVFbFw (2023)
[2] Mitigations are attack surface, too https://projectzero.google/2020/02/mitigations-are-attack-su... (2020)
Toro does not provides that separation. However, I was having some thoughs about running the user app in ring1 to provide some sort of separation whereas the kernel runs in ring0. However, in that case, we may end up in the current user/kernel level separation of general purpose OSs.
so, generally at the edge (gateways, shims, protocol boundaries)
file sharing is complex too it seems
would be good to see a benchmark or something showing where it shines
What use cases would Toro fit? pros and cons ?