Kata Containers sidestep this issue by running Kubernetes containers as lightweight VMs. Same for Firecracker. Did the team consider microVM solutions?
I'm not sure how much easier to deal with the MicroVM model would be. I think it would be great to add the additional layer of security that gVisor and Co have. There's also benefits of being able o run a special kernel per workload. On the other hand, you do gain some complexity by having to run a separate kernel per workload, and having to inject all the sidecars (SSH, secret manager, etc..) into the VM.
From a performance perspective, there's a lot of unknowns. Up until relatively recently, virtfs did not exist, so you had to carve off a block device. You can't do things like share a page cache and dedupe inodes and dentries AFAIK with MicroVMs. Then, on top of that, there are still security boundaries we need to enforce, like network filtering. If you do this with VMs, you need to do it at the packet layer versus the connection layer, which is much more expensive. There are others...
Lastly, it's a question of maturity. "full blown" VMs are very mature. On the other hand, containers have been maturing for upwards of a decade now. We'll probably explore MicroVMs further in the future, but for now, Linux containers are the tool of choice.
I work on several large Kubernetes clusters each running diverse jobs and this would be a huge BENEFIT. The Linux virtual memory (VM) subsystem regularly locks under heavy contention (e.g. one process performing a ridiculous amount of local, buffered I/O, dirtying a ton of pages, while another process struggles to page back in its executable code), leaving random processes frozen for seconds or minutes. (Default frozen lock failsafe triggers after 120s, but when things are really bad I've seen processes frozen for much longer.) It's a persistent source of alerts and complaints from people. It was much worse last year (2019) owing to some kernel regressions, but it's beginning to creep back up as the clusters see more production use so I'm not looking forward to 2021.
Kubernetes pods rarely need to share filesystems between themselves--at least not for heavy usage--and if they do it's usually via NFS or something similar. So MicroVMs really only leave you with the fixed cost of duplicating kernel data structures, which given the amount of memory on most nodes isn't that much. And there's performance potential here precisely because processes in different VMs aren't contending for as many of the same locks, etc. The tradeoffs very much mirror those of multi-process vs multi-threaded and monolithic vs microservice architectures.
Realistically, though, I can see some performance degradations due to the indirection needed for accessing local storage, except where you can make use of hardware passthrough. But I'll take a few percentage losses in runtime costs over frozen processes.
Linux just isn't as robust with these heavy, multi-tenant environments. That's what operating systems like Solaris excelled at. It's kind of ironic, but not really because that's typically how these things play out, unfortunately :( People are attracted by the ease and low-cost of running popular Linux-based solutions, but then try to bend them to run the kinds of workloads and architectures the older systems were designed and optimized for. MicroVM architectures are better suited to Linux' strong points.
And that's all before we ever consider security. User namespaces are just a band-aid over the problem of Linux security. The real issue is the endless parade of kernel exploits. That's where seccomp comes in. But expecting SREs or even developers to properly seccomp-jail all the myriad containers deployed--homegrown and especially third-party--is not realistic. The whole conceit of containers is to lower the barrier of entry to writing and deploying applications at scale while pretending we can still protect people from themselves. If people couldn't figure out how to run multi-tenant before containers (that is, leverage traditional APIs, process model, and networking stack), how could we ever expect them to do any better with even more complex infrastructure? We already tried all that with SELinux. seccomp is better than SELinux because it puts the application developer in control, who is best positioned to know when, where, and how privileges are needed (assuming they know at all). Moving that work back to, effectively, the system administrator isn't going to turn out any better than SELinux did, or present-day container security for that matter.
I wonder if we will see kernel namespaces that would let you run specialized kernel for each container just like a VM.