Hacker News Reader

> Bare metal isn't the same, and who wants to dedicate a whole machine to one process?

1) I actually do that at work. A few bare metal machines each have to run just one large-ish process. If I had the time I'd really invest in a unikernel type setup just for the simplicity of it.

2) if you're using any cloud hosted vm to run your apps as the post describes you could benefit a lot from a unikernel setup. Haven't used aws et al much the last few years but at my last gig we hosted everything in their own vm,which would've been great to slim down hard.

The alternative is not between bare metal and paravirtualization. Hardware assisted virtualisation is a thing. The hypervisor can actually be very thin.

I think what you are describing is actually an argument in favor of unikernel. The idea is to get the best of both world. You get the isolation guaranty provided by virtualisation without the cost of a full OS running on a full OS and with the added benefits of compiler optimisation running on the whole stake at once.

Intellectually it makes a lot of sense. The question is: can it actually work without creating a lot of frictions during development and deliver these theoretical gains.

The host interface is paravirtualized. Hardware assisted virtualization only covers the CPU in most cases. All the hardware and I/O is going to be using paravirtualized devices (virtio or similar).

There is no inherent reason why that interface is going to be more secure than the syscall interface. If the goal is security, there are easier ways of hardening that without reaching for virtualization. Virtualization itself doesn't really buy you anything there; we only consider VMs more secure due to how existing implementations work, not due to anything intrinsic.

> Hardware assisted virtualization only covers the CPU in most cases. All the hardware and I/O is going to be using paravirtualized devices (virtio or similar).

Well, no. IOMMU is a thing. You can do passthrough of I/O devices is you need to.

> There is no inherent reason why that interface is going to be more secure than the syscall interface. If the goal is security, there are easier ways of hardening that without reaching for virtualization.

There is no question that running a software in a VM is safer than running it straight on the kernel. The isolation enforced is far stronger.

You could legitimately ask if it’s safer than running it inside a container. That’s a good question especially now that the orchestration surrounding containers is better.

A unikernel keeps the advantage of discarding plenty of attack vectors just by virtue of every unnecessary part of the kernel not being there however.

> we only consider VMs more secure due to how existing implementations work, not due to anything intrinsic.

It’s weird argument. We do indeed consider VMs more secure than running things directly on the kernel because the way they work intrinsically means are.

You can certainly harden systems in different way and enforce isolation though different means but VMs have the advantage of being a fairly old and well known technology with good toolings around them, especially in the entreprise world.

> Well, no. IOMMU is a thing. You can do passthrough of I/O devices is you need to.

That's not virtualization. That's explicitly not virtualizing that device. I don't think any cloud vendors do that for anything but perhaps GPU acceleration. It doesn't make much sense for a unikernel - would you really dedicate a NIC to each app?

> There is no question that running a software in a VM is safer than running it straight on the kernel. The isolation enforced is far stronger.

There is no magic extra isolation. You think there is because of what we've built with VM technology. The underlying hardware doesn't provide extra isolation.

I recently built my own hypervisor from scratch for ARM64. There is no magic "isolation on" bit. On ARM64, there are three execution levels: EL2 (hypervisor), EL1 (guest), and EL0 (userspace). Running stuff in userspace (EL0) in a VM is identical to running it in userspace outside a VM (also EL0), except for whatever VM features you've turned on. These are things like second stage paging, trapping certain system operations, etc. Effectively none of these have any impact on EL0; they are all there to allow you to run a guest kernel at EL1, not to isolate it more than userspace already is. In fact, in non VHE mode, your host OS kernel also runs at EL1 - kernels without VM support just drop to EL1 from EL2 on startup and effectively run as "uniguests" with no hypervisor.

Intel works a different way, but that's just implementation technicalities; the security position is the same in practice.

We consider VMs to have better isolation merely because we've done a better job at having a smaller attack surface in our paravirtualized interface designs than we have with our syscall interface designs. Nothing more. It's all in the software. It has nothing to do with the actual VM part.

> A unikernel keeps the advantage of discarding plenty of attack vectors just by virtue of every unnecessary part of the kernel not being there however.

But the kernel is there. It's right there outside the unikernel. You just have a smaller attack surface because its paravirt hardware interfaces are better designed than its syscall interfaces.

> You can certainly harden systems in different way and enforce isolation though different means but VMs have the advantage of being a fairly old and well known technology with good toolings around them, especially in the entreprise world.

Now you're getting somewhere. Yes, VM tooling makes VMs more secure. As I said, it's purely a result of how we use VM technology. It's not intrinsic.

> That's not virtualization. That's explicitly not virtualizing that device.

So what? You are still running on top of the hypervisor. It’s entirely doable from a unikernel.

> It doesn't make much sense for a unikernel - would you really dedicate a NIC to each app?

Of course it makes sense. The point is if you need the performance of a dedicated NIC you can get them, same things for the other I/O interfaces. Need a dedicated disk array with full performance? You can get it.

> There is no magic extra isolation. You think there is because of what we've built with VM technology. The underlying hardware doesn't provide extra isolation.

I think there is because there is. When you write « what’ve built with VM technology », that’s what that means. It doesn’t matter that you could in theory do the same on the kernel because there is no underlying mechanism at the hardware level. In the actual world, VM are safer.

> But the kernel is there. It's right there outside the unikernel. You just have a smaller attack surface because its paravirt hardware interfaces are better designed than its syscall interfaces.

If you can’t even call the interface because support is not there in the unikernel it’s not part of your attack surface.

But you are once again using a completely flawed argument. It doesn’t matter that syscall interfaces could be as safe as the one provided by the VM because they are not. This is not an argument about potential. You are completely missing the point because you live in a fantasy.

> So what? You are still running on top of the hypervisor. It’s entirely doable from a unikernel.

Is it? Does this unikernel have support for all the hardware that the Linux kernel does?

Once you're dealing with real hardware, 90% of your project becomes drivers, and 10% the actual kernel bit. No niche kernel can compete on that front.

> Need a dedicated disk array with full performance? You can get it.

Meanwhile you're still paying for VM overhead (it's not zero) for no good reason...

Again, this thing's selling point is "it runs in the cloud", and cloud vendors don't do passthrough. If you're running your own metal, you're better off with a sandboxing approach that doesn't needlessly involve a full VM.

> In the actual world, VM are safer.

There are many sandboxing technologies not built on VMs with good track records (or even built on VM modes but much lighter than a full virtual PC like cloud vendors use). Do you have statistics on how VMs are safer than these kinds of technologies? OP is only comparing containers and VMs. There's plenty of more secure technologies, like gVisor (that has a backend that uses KVM and one that does not, and is in both cases lighter weight than a full x86 virtual machine).

> If you can’t even call the interface because support is not there in the unikernel it’s not part of your attack surface.

If you can't even call the interface because support is not there in the sandbox layer, it's not part of your attack surface.

> But you are once again using a completely flawed argument. It doesn’t matter that syscall interfaces could be as safe as the one provided by the VM because they are not. This is not an argument about potential.

Again, look at sandboxing technologies. The kind that use either full on syscall wrappers or at least very strict whitelists.

There have been plenty of VM escape bugs going around. Syscalls have a larger attack surface because they do more stuff. Start filtering all that with a strong sandbox and you can achieve similar levels of security to VMs. There's no magic thing about VMs that makes them safer.

> You are completely missing the point because you live in a fantasy.

I guess Google is living in a fantasy too, seeing as they've been running untrusted customer code on App Engine by using sandboxes, on top of kernels shared between multiple customers and also internal code, with no VMs in sight (at least originally; they may have moved to the KVM backend of gVisor now, but again that's still not a full blown virtio PC like the thing this unikernel runs on).

When I studied in Uni, it seemed like there were two different paths to reducing the number of copies and system calls in handling a network request.

1. Move the network stack to userspace. Then the code running in userland can get at the request without a copy.

2. Move the running code into the kernel. Then the code running in the kernel can get at the request without a copy.

Unikernals are a way to do 2. People just really don't want to pay for that syscall boundary.

I think the parent's point is that running a unikernel on paravirtualized hardware is actually neither: the entire VM runs in userspace as seen from the host, but you still have the data copy from the host's physical network interface to the paravirtual interface on the VM.

But you're paying for the virtualization boundary, which is usually more expensive than a syscall boundary. You're still a VM guest running under the host kernel, which is a comparable situation to a userspace process.

I don’t think App Engine shares a Linux kernel between users. That would be a surprising level of confidence in lack of Linux system call vulnerabilities that no other cloud vendor seems to have (see AWS’s Firecracker VM or Google’s own gVisor).

So at the end of the day, you’re still going to be dealing with a hypervisor. It’s also worth noting that at least AWS appears to hardware accelerate their paravirtualized interfaces so it’s not necessarily the case that IO operations are causing VM exits.

App Engine absolutely shares a Linux between users. They have a heavily audited sandbox with multiple layers of defence. Of course customers don't get to call arbitrary system calls, but it's also not a VM.

I don't know if they use Firejail or not but that sounds a lot like Firejail.

GCP indicates that App Engine standard environment at least uses gVisor[1] (i.e. KVM with emulated Linux syscall interface). The docs also indicate you can run custom binaries on most of the standard runtime languages[2] (not to mention being able to ship Go binaries), so I don’t see how they could get around allowing system calls.

App Engine must have used something other than gVisor originally since Go didn’t even exist when it launched, but I’m having trouble finding any concrete indication as to whether they shared kernels between multiple customers.

[1]: https://cloud.google.com/blog/products/containers-kubernetes...

[2]: https://cloud.google.com/appengine/docs/the-appengine-enviro...

gVisor's kvm backend is still experimental AFAIK, in favor of the ptrace backend.

I think the experimental warning was removed on the KVM platform a while ago?

They definitely shared kernels between customers (and with other internal stuff).

I don't know if they're using the KVM backend now, but if they are, that's definitely more recent. There was no virtualization involved when it all started.

Source: I'm a xoogler.