Hacker News Reader

Comment by muricula | Hacker News Reader

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/

View on HN

(I am co-author of the post)

System calls are important, but only one factor. The linked doc is an attempt to clarify and delineate various costs. There are number of platform options (the platform is what does syscall interception), and I don't believe any of them are 100x so to say "at least" is a bit disingenuous. You may have confused the "runsc-kvm" number with "using a VM". "runsc-kvm" is the system call performance of gVisor using the kvm platform, which is not a full VM [1]. In general the syscall cost in a VM depends entirely on the guest OS, since there is no VMEXIT for this operation.

VMs are a valid choice depending on your workload, and this is providing an additional tool that provides an easy control for containerized infrastructure. You can use what works for you. Native containers certainly work as well, but you'll probably want to consider additional security controls of some form if you're really running untrusted stuff in there.

[1] https://github.com/google/gvisor/tree/master/pkg/sentry/plat...

You're right, that's not the chart I wanted to see. I'm just dubious that reimplementing lots of the Linux kernel in Go while paying the cost of the ptrace interception is worthwhile. It seems like you're just adding a lot of attack surface (admittedly managed code > native code) with a large perf impact. Do you have any docs on how the kvm-runsc platform works? Skimming the files, I don't see some of the bits necessary for a bluepill style hypervisor, so I'm not sure why parts are named bluepill in there. I also don't see a lot of the linux kernel paravirt vdev code I would expect, and you seem to imply that you're not telling KVM to enable syscall trapping for the guest.

I'm not sure what you mean by KVM syscall trapping for the guest. The bluepill refers to the fact that the Sentry runs transparently in VMX non-root ring 0 and regular host ring 3.

I'm not sure what to provide re: docs -- the code is all there, reasonably documented and there are discussions on the public groups of how the KVM platform works. I feel a bit like you're coming in with a specific set of ideas and skimming files (e.g. the performance guide and the code itself) in order to confirm an existing understanding, but it's just not working.

I'd love more precise criticisms re: adding to the attack surface, but otherwise I'm not sure how I can help.

I'm very skeptical about the platform and don't have the time to devote to reading the codebase or having conversations as I would like. The TL;DR is that the syscall interception technique seems expensive and I wonder if you will write all sorts of logic bugs in the sentry broker. It seems like you folks care about security, and have some good ideas, but if you really care about hostile multi-tenant containers, why not stick the container in a VM and call it a day?

I replied in other comments but our talk at Next'19 [1] includes a story by one of our customers, which may help understand the use cases. In a nutshell, GKE Sandbox should allow sharing the resources of GKE Nodes (VMs) among multiple tenants.

[1] https://www.youtube.com/watch?v=TQfc8OlB2sg

These kernels and micro vmms (kata, firecracker, etc) are aimed at workloads that have already been containerized or are being deployed to container orchestration systems.

For some cases, having something that is compatible with kubernetes is worth the performance penalty, especially if your workload isn't syscall heavy.

where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.

Look at the syscalls chart. It's hard to tell exactly what the numbers are on the log log scale on the syscalls chart, and it's probably not the graph I want anyway, but it looks like their runsc-kvm platform clocks in at 1k ns/syscall while their ptrace platform looks close to 100k ns/syscall. The fact that it's log log is telling alone.

Missed that one. The gVisor calls are closer to 40k but the difference is indeed big.

It's just log, not log log. The chart generates from .csv hosted on the site [1] and the benchmark tools are all open.

The ptrace numbers are 20x and the KVM platform is actually lower than the Docker default case (though that doesn't mean everything is faster, as system call time is only one factor). As I note above, I think you're confused about what the KVM platform is -- it's not a VM.

[1] https://gvisor.dev/performance/syscall.csv

Reply on news.ycombinator.com