alexzenla

When building Edera (product from article), I also had the added problem of the virtual networking gap where I was bridging a 10Gbit NIC over a virtual interface, and I had weird performance bouncing between 3Gbit and the full 10Gbit. Luckily I had built networking drivers before and knew the complexities of it, and managed to profile it down to the virtual interface getting worst-case NUMA occasionally.

The part 2 is going to cover how we actually solved it, which involves every part of the system having knowledge. It's so easy to ignore but it has a massive impact on perf.

(CTO of Edera here)

Great point! We also try to factor that in as well.

Steven (the author) will cover that in part 2!

All of that is a yes, plus compressed memory is a big component of macOS.

When running on bare metal, the CPU performance is within 1%, so usually quite well! Hardest thing is I/O, but we do a lot to help with that too.

I'd say the limitation has been that sometimes we have to implement things by hand. But it has enabled us to do things that others can't achieve since KVM is a singular stack in many ways. For example, VFIO-PCI is largely the same across all VMMs, but we have true full control over the PCI passthrough on our platform which has allowed us to do things KVM VMMs can't.

I was more specifically referring to the fact that to implement threads in gVisor, it calls to the go runtime, which does make calls to clone() (not fork()), but I see the pushback :)

I think it's a small distinction. fork() itself isn't all that useful anyways.

However, consider reading a file in gVisor. This passes through the IO layers, which ultimately will end up a read in the kernel, through one of the many interfaces to do so.

This is why you can't build these microVM systems to just do isolation, it has to provide more value than that. Observability, policy, etc.

This is the thesis of our research paper here, a good middle ground is necessary: https://arxiv.org/abs/2501.04580

The reason why virtualization approaches with true Linux kernels is still important is what you do allow via syscalls ultimately does result in a syscall on the host system, even if through layers of indirection. Ultimately, if you fork() in gVisor, that calls fork() on the host (btw fork() execve() is expensive on gVisor still).

The middle ground we've built is that a real Linux kernel interfaces with your application in the VM (we call it a zone), but that kernel then can make specialized and specific interface calls to the host system.

For example with NVIDIA on gVisor, the ioctl()'s are passed through directly, with NVIDIA driver vulnerabilities that can cause memory corruption, it leads directly into corruption in the host kernel. With our platform at Edera (https://edera.dev), the NVIDIA driver runs in the VM itself, so a memory corruption bug doesn't percolate to other systems.

This is a big reason for our strategy at Edera (https://edera.dev) of building hypervisor technology that eliminates the standard x86/ARM kernel overhead in favor of deep para-virtualization.

The performance of gVisor is often a big limiting factor in deployment.

Not sure why this says East coast, I am also affected here in Seattle.

Hi! I’m the CTO of Edera and discovered this bug with my colleague Steven!

The story of this bug is interesting. We were both up late at night working on GPU support on the Edera platform, and we had just pulled an NVIDIA container image. What should have resulted in a temporary directory of tar files for OCI layers was filled with NVIDIA library files! We were both super confused until I had an “oh god no” moment and realized this happened.

We kicked right into action on responsible disclosure.

I can answer any questions, but I want to send a huge thank you to our team for working together on this and to Astral for being wonderful to work with!