Unikernels were hard. key word: were
ghuntley.com
ghuntley.com
> With a unikernel, the [observability] surface is much smaller. If the functionality isn't in the application (the operating system), then the [SRE] is screwed. There's no next hop.
Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value. The article mentions some nice things for observability ("structured logging, OTel") but so much gets thrown away.
The tools exist, and in a container world no one should be running those tools directly anyway, it is all about rootless immutable containers, ideally without OS layers for deployment.
There isn't a Linux server taking care of a couple of services.
There are services deployed as Kubernetes pods, or serverless language runtimes across a cluster abstraction, running directly on top of type 1 hypervisors, or minimal kernel images enough to power distroless container images.
Also, this argument only works if you just broaden the definition of “OS” to “anything that runs that isn’t your apps”.
Sounds like Plan 9. Which would put that around 30ish years ago. Mainframe OSes can probably make similar claims even earlier.
Not that many people keep large distributed workloads anymore, and SLURM-like schedulers on top of regular operating systems were deemed good enough for supercomputers.
The Cray T3D had a really nice distributed services model (really kind of a distributed exokernel) that would be worth emulating
on some machines there was an interactive ganged run facility like 'mpirun'. I think even this distinction where there are 'login' or 'service' nodes and anonymous runners would be at least a little structure. I guess you could argue that k8s is this, but it kind of falls short and seems to require a lot of continuous propping up.
I worked on a single system image project that didn't ever get shipped. it turns out that a syscall proxy layer isn't that difficult to implement, and that leaves you with an undisturbed user space.
anyways, there are a whole host of design options that don't look like k8s or slurm. and having spent some time recently on non-hyperscaler clusters, that bottom level of bringup (provisioning, networking management, reporting, fault tolerance, distributed filesystems) is really a nightmare. I think we lost a lot of traction there by raising generations exclusively in someone else's operational context.
Trivial to implement. I have a full custom (from scratch written) debugging/benchmarking implementation written both for Linux and two of my custom kernels, both of which are REALLY unique when it comes to architecture. My custom debugger also supports numerous inline scripting languages.
> Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value.
But doesn't their value lie in getting you access to the core of the OS that you wouldn't be able to touch otherwise? What if your network stack was a library you could instrument just like you do with the rest of your app code?
Also, I don't need a tool to tell me I'm being punished by the IO or CPU scheduler if I am the only one using the CPU and IO :)
On the plus side: it's a unikernel. You compile it and your application together and get a binary. It has support for running on virtual hardware (https://docs.zephyrproject.org/latest/hardware/virtualizatio...) and virtio is the new bios, right?
On the minus side: I think porting "traditional" software - say Nginx - to it will be agonising. And I can't make any guarantees about its performance in the same way that Alpine Linux container images are not necessarily fast.
But there's an entire toolchain, debugging story, drivers etc. for something that compiles to tiny and starts up basically instantaneously. Surely that's got to be useful for something?
I think this is basically a solved problem with enough agents and enough compute now
Has the math changed? Maybe. I remember the days before multitasking and protected memory. Things were faster, but more fragile. But these days there are so many things happening that they are, on some dimensions, just as fragile.
The real cost is going to be hardening, and maybe LLMs can do that as well.
So you get a type 1 hypervisor, which technically is like a micro kernel, which then boots whatever else on top including language runtimes, or chiseled containers.
Unikernels ideas aren't much different from some embedded deployment scenarios, with the application, language runtime and OS all linked together into the device firmware.
Naturally on desktop computing is another matter.
~$0.005 CAD/h for a small network-enabled VM with 4MiB of RAM and no disk.
Is this kernel the same one still?
The kernel is mainly the same - just modifications for running under Firecracker.
I might be interested in this because that is cheap, but what exactly are the networking rules/costs? ip4/6, bandwidth, egress, throughput, connection limits?
None yet - this is a beta so bandwidth pricing will come later.
The Zynq UltraScale+ parts pair a hard ARM CPU with FPGA fabric. You can progressively isolate fast path and high security components in logic. LLMs are getting quite good at using synthesis tools.
https://www.amd.com/en/products/system-on-modules/kria/k26/k...
In an oxide episode there were some mentions of reading off data lines but I just don’t think that’s practical.
The reduced attack service is cool but not at the expense of my visibility and liveness of the system
You could make the argument that Rust w/ its memory safety is a candidate, but w/ Rust it's still entirely possible and fairly easy to break out of that. And Rust/Cargo applications have a habit of using a bazillion third party deps that you then need to keep a close eye on.
people always talk about losing the excellent debugging facilities that linux gives you. if a service crashes in a big cloud environment what is that you do now that works so well. do you think its intractable to implement that in a single-process kernel?
Use Rust and the chances of an application overflow drop off a cliff. The debug-ability also increases the surface for bugs themselves. The majority of cases we're talking about chips that can comfortably be hardware-simulated. And/or all you need is some way for a dev machine to edit memory and an AI can work through the night to explore a configuration space much larger than you would ever have.
This is one reason I'm so bullish on Lean4 as a general programming language
It's now essentially proven that expressive, strong type systems like those of Rust present an advantage for LLM assisted coding
Lean's type system, in which types are first class, fully programmable objects takes it so much further. Particularly when it comes to embracing dependant types
But you know what else is easier with LLMs? Properly hardening your Linux system. "Patch every week" is the recommendation from upstream because the kernel security team has to consider every possible deployment configuration and environment. If you implement all the KSPP recommendations, use a config and kernel command line that makes sense for your actual application, and use a properly configured MAC LSM, you would not have been affected by any of the big blockbuster vulns of the last year.
So if you can tell an LLM to port a library in a loop and feel confident in the results, surely you can feel equally confident telling the LLM to harden your Linux environment.
n.b.: I am not endorsing the idea that just throwing LLMs at security problems is actually a good answer, I'm just saying that the case the article makes isn't actually comparing like for like.
I’ve started looking at them myself, and have been comparing them to mature VMMs like Firecracker, and asking myself where each piece of my stack might be best run.
Virtual machines and containers are no longer an effective isolation mechanism when AI is involved. VM breakouts are becoming trivial. So everyone should be considering how to bake better security into the their runtimes.
How exactly?
They are not. When discovered, they are still big news. With few exceptions, businesses (and other institutions) continue to treat the cloud providers' VM-as-a-service offerings as secure. AI can aid the defenders too.
> So everyone should be considering how to bake better security into the their runtimes.
If VM isolation is not trustworthy, my first concern would be whether the other VMs on the physical host machine might do something malicious and break into my VM.
That's why I never really liked talking about "reducing attack surface" that much because folk inevitably turn to lines and code, which while reducing is good, just simply doesn't communicate what the biggest problem truly is.
Vuln exploitation is the number one entry point for data breaches and os command injection is the number one CWE in CISA Kev from last year.
System intrusion was repeated something like 64 times in last year's DBIR.
The operating system itself is literally the problem as it's inherently meant to run many different programs whereas unikernels only run one.
Unless you rely on security through compartmentalization. See: https://qubes-os.org
Yes, in theory you have a smaller attack surface, in practice its not exactly true.
Yes, you throw away a lot of tools that you don't need, which means that sideways traversal is much much harder, it doesn't actually stop your front door being smashed down.
You are also on the hook for detecting and updating any ported library. This sounds easy right? just look for the CVEs and then re-build the image?
Ok but what version of the ported library is in your unikernel? in the porting did it have the same bug? Did the LLM cock up the porting, did
Basically you are on your own. So saying "unikernels! you're secure now" is at best misleading.
These apps tend to do kernel bypass anyway for net i/o after hard scrubbing the os to remove as many interrupts and other superfluous jitter the oms does not need.
However, I believe the low level kernel bypass code (eg dpdk, libfabric) depend on Linux to talk to the hw in a disciplined way.
Anybody know better?
A decently faster oms this way could be a practical alternative to fpgas we sometimes see in that space. (hw would be co-located of course)
I saw a post that said roughly "I thought computing was doomed to be 'Linux, Python, and HTTP' for the rest of my life. Now we're free". That's it. Let it sink in. We're free. We can make the computer do whatever we want.
Cast off the yokes back-compat and history
I used to regard V8 Isolates as a best possible sort of technology, with userlands juggling lots of processes.
Seeing netlify & unikraft switch to microvm's and have such a huge speed up was a bit of an awakening for me. Those are really fast start times! https://www.netlify.com/blog/edge-functions-firecracker-micr... https://unikraft.com/customer-stories/edge-functions-netlify...
Intuitively, I think I have some appreciation for how much silicon has been poured into virtualization. Its always seemed like a "yeah but you could avoid those costs by not doing that" but I'm more receptive to the idea that these might in some cases be really good ways to get some of the isolation workloads demand with the hardware helping us out, these days. There's so much securing for vm's, and maybe it's just easier than trying to secure in userlands: let the hardware help.
Long time interest in microvm's but they felt heavier weight than I wanted. Now it feels like maybe they might actually in some regards in some ways be lighter weight than managing workloads yourself in userland. Maybe. I dunno. Interesting times, i'm open to it.
Edit: I just chatted with Astra Pro some about these ideas, if anyone wants some sense material to chew on. https://chatgpt.com/share/6aca924c-8e64-83ea-a5c3-aae08b2cdf...
I'd wager most of the services running on the interwebs are web servers, database servers, inference servers etc that don't change over their lifetime really. They're just doing the same thing all day long every day.
If you're doing stuff like what I'm doing for work right now, which is, yeah, multiplexing potentially oodles of user-submitted jobs, and those jobs are best expressed as distinct images or containers, then yes, managing start times is absolutely imperative in improving utilization/occupancy and therefore reducing costs.
But I'm not convinced that's a typical scenario, not typical enough to drive enough time and money investment in this space maybe?
Also there ain't currently no real "hypervisor" for the (NVIDIA) GPU. Not practically anyways. And that's arguably where we need it the most. Or at least I do, for Day Job(tm).
So unikernels and microvms may have to lean on other arguments for adoption: security and simplicity-to-reason-about might be those...
The application server has been the only way to build systems, because we couldn't imagine running lots of secure processes. But what if we could? And... We can. Probably. If we tried.
The simplicity and security from the processes being more small workload processes, from not having some webserver treating each request as more semi anonymous internal non-disrinct workload could be really interesting. And it could let us manage systems much better. Maybe.
I come from security so I'm more disposed towards the benefits thereof, however, being a person that has had a commercial and open source unikernel presence for a while now - "ease of use" is probably the largest selling point that I see from most of our user base. There is nothing out there that gives you the performance, security and cost benefits from shipping your unikernels to AWS, GCP, etc. in seconds. Not lambda, not cloudflare, not any of the millions of microvm providers. You quote netlify right here (who actually don't use unikernels per-se but small little linux vms (big difference!)) but why would you even use them when you can do it cheaper by shipping straight to ec2?
That's what unikernels allow you to do.
Maybe a good halfway point is to still have a unikernel but the libraries for common stuff like networking or fs are already written according to a standard and plug and play and reusable across applications?
The point that makes this different from an OS is that there's no shared service, no syscalls to do that stuff -- just subroutine calls -- and no timesharing (except at the hypervisor level). It's just a single runtime running on hypervisor direct against the virtualized hardware.
Because, yeah, all this work has happened in hardware in the last 30 years to make that possible, but we still treat the operating system as the best unit of resource sharing. When in fact it's kind of a jack of all trades master of none. If you're booting a whole linux kernel -- with its giant framework built to co-tenant a pile of applications and users -- just to run one process, there's definitely something to look at there. Even if the unikernel space itself is relatively immature.
You don't need to emphasize the "vibe coded every time you need them" thing. That's not what anybody was being talked about in the article. The "vibe coding" piece is pointing out that the missing pieces in the ecosystem can be more rapidly filled in now because of agentic tooling.
Well, here is your actual OS then. You named it "hypervisor", and changed the system interface, but it's still there, revirtualizing the access to hardware and ensuring separation of different users/applications/unikernels.
The point is all the other things: total number of lines of code, permissions and security, memory management, drivers, process mgmt, it all changes.
I do know that AMD has done more in this space. When I worked at Google there were (other) people on the Stadia team doing this. And some open source bits out there that I've seen, including from AMD.
Unfortunately, my real world work... and most real inference etc work others are doing... is on CUDA/NVIDIA.
Or if you're more familiar with embedded, like a BSP or RTOS
Unless you're literally running a single application on a dedicated piece of hardware in which case... yeah, sure, you can do the OS-as-a-library schtick. But you could do it 10 years ago, too.
With a capability-based microkernel you get to assign ownership and resources to a very granular level that increases robustness against crashes and protection against attacks to the core system policy. A unikernel can give all of that but I feel like then it will be worse to program against rather than a microkernel.
So overall I don't see any advantage of unikernels. They don't help you write more correct programs IMO and the performance gains are limited.
IME you can eliminate tons of issues using language-based safety. That just wasn't possible before Rust without taking a perf hit. And you can still use vm in a unikernel for e.g. guard pages, read-only segments, demand paging etc.
I'd say the main advantage of the unikernel approach is the ability to be very intentional about where/how you introduce complexity. For a certain class of high-performance systems software, that has direct implications on reliability and performance.
Similarly, a filesystem's just a data structure that happens to be on disc. You can mmap your harddrive and let the host hypervisor handle swap, preemption etc.
It's similar to a process contract, but with slightly different boundaries
This is all unnecessary overhead and complexity, as the OS doesn't really mediate hardware access in this case.
But you're right on some points, it's not really more secure, but there's a hell of a lot less complexity where things can go wrong.
The only advantage is that cloud services provide VM tenants so you are already stuck with a VM substrate and you want to minimize layers from that point on.
However, as a practical matter, the design of the Linux kernel is grossly insecure and basically impossible to make even remotely safe as evidenced by the unending sprawl of LPEs. Hypervisors are generally designed and configured in a way that is only mostly insecure instead of grossly insecure; a bad is better than terrible situation.
However, there is nothing stopping you from designing a kernel in a similar way and providing shared tenant processes with similar guarantees. You just need to port your applications to the new operating system, but you would also need to port them to a unikernel anyways if you wanted to go that route.
The only free action is lumping your applications with the full OS and dumping it onto a hypervisor. Anything else has porting work which is why the multi-tenant VM model is advantageous at all in the first place.
Still, there's a whole pile of stuff there that many applications don't need. If you truly can run the entire of the OCaml runtime and standard libraries etc without it, and you're happy writing your apps in OCaml... MirageOS has always felt winning-ish to me as a general idea.
And to TFA's point, now with LLMs the relative exotic nature of OCaml as a platform need not be a huge hindrance. I like the language, but have only dipped my toes in. Definitely tempting to try writing apps on it w/ MirageOS now. Though I doubt I'd convince anybody to pay me to do that...
A lot of comments here about observability. Well UKL solves this nicely because you can run regular userspace processes in the unikernel, thus you can (optionally of course) run a shell, perf, and anything else you need to observe what the machine is doing.
Linux is about 40 million lines of code, and even if you disable almost everything optional, it's hard to get a usefully bootable Linux below about a megabyte.
The seL4 kernel is about 9000 lines of code, a code base that's 4000 times smaller, and I think it compiles to about 80KiB, about 12× smaller than Linux.
I elicited a small unikernel web server from Claude the week before last, and, not counting lwIP, it's 385 lines of code. The built unikernel image, which does include lwIP, is 58K, about 17× smaller than Linux.
The KokalOS kernel I've been working on over the last week, originally by refactoring the unikernel web server, is currently about 900 lines of code (65536× smaller than Linux) and compiles to just under 10KiB (128× smaller than Linux), but it's currently just running a CLI and a web server with a single Ethernet card. I anticipate that in a finished state it will probably be close to the size of seL4.
Type 1 hypervisors running language runtimes on top, for serverless computing, aren't much different from the unikernels vision.
You're Linus arguing against Tanenbaum. Even if you win, you still lose.
https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
Looking at what's out there, I think GenodeOS should be mentioned for application-specific systems. They can even build some of their stuff on seL4 if you want.
I'll also add there's been unikernel projects that aren't in exotic languages. Even C++ (IncludeOS?). There's probably one in Rust by now. If not, it could probably build on Redox's components.
I was a swing vote and you lost my support in the first debate
What I care most about is benchmarks. I see AI rewriting its harnesses for the hardware it’s on and getting faster tokens/sec, and I can see that as the future. I think, okay we can go deeper
Nothing in this article addressed results, its just breaking every known convention in the scariest and most inconvenient way possible while claiming that the attack surface is smaller when its actually unknown with now limited debugging capability
I could overlook that if I got more access to RAM this way and wildly improved performance
Is it a good thing to have so many processes that the user doesn't understand and often isn't even aware of?
If it's a security concern now, think of how bad of a problem it will be after AI is writing most programs that people use.