All the major clouds use vm boundaries and separate K8s clusters between customers. Microsoft was similarly bitten a few years ago with one of their function products that expected K8s to be the primary security boundary.
Maybe I missed something in the article but where are they expecting any hard guarantees. If there is a model being trained for example (running arbitrary code) where does a multi K8 tenency play?
The main issue I see is all internal network communication was trusted once behind the proxy/firewall (Istio) but I probably just don't understand k8 clusters too well.
https://istio.io/latest/docs/ops/deployment/architecture/
Being able to run as the Istio user (1337) renders the proxy itself moot right ?
One needs to look into things like VirtualClusters to even begin to consider hard multi-tenancy with potential hostile tenants(https://github.com/kubernetes-sigs/cluster-api-provider-nest...). That is just about the control plane. It doesn't even touch the data plane.
How secure that is even with the extra layer, I do not know. Even in the VM land we have seen crazy VM escape exploits over the years.T
Kubernetes itself is very complex. The "who needs a UI when you have configuration files and an API?" approach makes it even more opaque to the people who often end up responsible for it. The landscape changes very rapidly.[2] I'd trust Kelsey Hightower to set up a secure multi-tenant deployment, but probably not anyone else.
Is it not practical to deploy clusters on top of virtualization? That should make efficient use of hardware while still giving each tenant their own cluster, therefore providing stronger isolation than the typical Kubernetes configuration tends to.
[1] I am specifically referring to a Kubernetes deployment where different customers are running custom code on the same underlying hosts. Using Kubernetes to host a service that is multi-tenant at a higher level is not something I would recommend, but it's not as immediately dangerous as a model where customers run container-level custom code.
[2] This is not surprising for a relatively new technology, especially one that's as paradigm-shifting as Kubernetes was. But most people are not going to rearchitect and redeploy everything every six months just because the Kubernetes developers decided to replace a pod security or network security model with a non-backward-compatible alternative again.
No it is not. Not in the way they are using it.
There are two main use-cases. One is multiple teams, in which case they are bound by their company's policies and guardrails
The second is multiple customers. But that also assumes they have no direct access to the cluster. A vendor would, instead, deploy multiple instances of a workload; the customers would not.
Straight from the horse's mouth: https://kubernetes.io/docs/concepts/security/multi-tenancy/
There's also nothing that says that multiple clusters need to be expensive, if they are sized right. They can be as small as you need both in number of instances and instance size. The overhead here is the control plane but, for small clusters, the resource requirements are similarly small.
That said, if hard multi tenancy is what you need, then you need to use things like this: https://github.com/kubernetes-sigs/cluster-api-provider-nest...
(for the control plane - you still need to worry about the data plane)