Building the largest known Kubernetes cluster

I am extremely Not A Database Person but I understand that the rationale for Kubernetes adopting etcd as its preferred data store was more about its distributed consistency features and less about query throughput. etcd is slower cause it's doing RAFT things and flushing stuff to disk.

Projects like kine allow K8s users to swap sqlite or postgres in place of etcd which (I assume, please correct me otherwise) would deliver better throughput since those backends don't need to perform consenus operations.

https://github.com/k3s-io/kine

it's not really bottlenecked by the store but by the calculations performed on each pod schedule/creation.

It's basically "take global state of node load and capacity, pick where to schedule it", and I'd imagine probably not running in parallel coz that would be far harder to manage.

It says in the blog that they require 13,000 queries per second to update lease objects, not that 13,000 is the total for all queries. I don't know why they cite that instead of total, but etcd's normal performance testing indicates it can handle at least 50,000 writes per second and 180,000 reads: https://etcd.io/docs/v3.6/op-guide/performance/. So, without them saying what the real number is, I'm going to guess their reads and writes outside of lease updates are at least much larger than those numbers.

For those not aware, if you create too many resources you can easily use up all of the 8GB hard coded maximum size in etcd which causes a cluster failure. With compaction and maintenance this risk is mitigated somewhat but it just takes one misbehaving operator or integration (e.g. hundreds of thousands of dex session resources created for pingdom/crawlers) to mess everything up. Backups of etcd are critical. That dex example is why I stopped it for my IDP.

> It makes me sad that to get these scalability numbers requires some secret sauce on top of spanner, which no body else in the k8s community can benefit from.

I'm not so sure. I mean, everything has tradeoffs, and what you need to do to put together the largest cluster known to man is not necessarily what you want to have to put together a mundane cluster.

It's possible I'm talking out of my ass and totally wrong because I'm basing this on principles, not benchmarking, but I'm pretty sure the problem is more etcd itself than boltdb. Specifically, the Raft protocol requires that the cluster leader's log has to be replicated to a quorum of voting members, who need to write to disk, including a flush, and then respond to the leader, before a write is considered committed. That's floor(n/2) + 1 disk flushes and twice as many network roundtrips to write any value. When your control plane has to span multiple data centers because the electricity cost of the cluster is too large for a single building to handle, it's hard for that not to become a bottleneck. Other limitations include the 8GiB disk limit another comment mentions and etcd's hard-coded 1.5 MiB request size limit that prevents you from writing large object collections in a single bundle.

etcd is fine for what it is, but that's a system meant to be reliable and simple to implement. Those are important qualities, but it wasn't built for scale or for speed. Ironically, etcd recommends 5 as the ideal number of cluster members and 7 as a maximum based on Google's findings from running chubby, that between-member latency gets too big otherwise. With 5, that means you can't ever store more than 40GiB of data. I have no idea what a typical ratio of cluster nodes to total data is, but that only gives you about 307MiB per node for 130,000 nodes, which doesn't seem like very much.

There are other options. k3s made kine which acts as a shim intercepting the etcd API calls made by the apiserver and translating it into calls to some other dbms. Originally, this was to make a really small Kubernetes that used an embedded sqlite as its datastore, but you could do the same thing for any arbitrary backend by just changing one side of the shim.

It's totally possible to run tens of thousands of QPS on etcd if your disks are NVMEs (or if you disable fdatasync which is not recommended). If you use kine+cockroachdb or tidb you can go even higher which is what I'm guessing is equivalent to their spanner setup.

There was a blogpost about creating an alternative to etcd for super high scale kubernetes cluster. All code was open too. It was from someone named Benjamin I think but not sure.

I’m not able to find the blogpost but maybe someone else can!

I’ve seen some talk of replacing etcd with FoundationDB, which could yield similar improvements.

That is 100k vs 130k for Google’s new announcement. I can’t speak as to whether the additional 30k presented new challenges though.

That's simulated using kwok, not real.

> Unfortunately running 1M real kubelets is beyond my budget.

THis is a PoC not backed by a reliable etcd replacement.

fuse based filesystems in general shouldn’t be treated as production ready in my experience.

They’re wonderful for low volume, low performance and low reliability operations. (browsing, copying, integrating with legacy systems that do not permit native access), but beyond that they consume huge resources and do odd things when the backend is not in its most ideal state.

Same here. Non Kubernetes project originated control plane components start failing beyond a certain limit - your ingress controllers, service meshes etc. So I don't usually take node numbers from these benchmarks seriously for our kind of workloads. We run a bunch of sub-1k node clusters.

Same. The control plane and various controllers just aren't up to the task.

Meh, I've had had clusters with close to 1k nodes (w/ cilium as CNI) and didnt have major issues

>You are not Google.

It's literally Google coming out with this capability and how is the criticism still "You are not Google"

I work for a mature public company that most people in the US have at least heard of. We're far from the largest in our industry and we run jobs with more than that almost every night. Not via k8s though.

> You are not Google.

100% agree.

People at my co are horny to adopt k8s. Really, tech leads want to put it on their resume ("resume driven development") and use a tool that was made to solve a particular problem we never had. The downside is now we now need to be proficient it at, know how to troubleshoot it, etc. It was sold to leadership as something that would make our lives easier but the exact opposite has happened.

> You are not Google.

You think they are just running it for fun? It's literally non-Google customers who wanted this as was explained in the article

This. I may not work with AI training workflows, but I struggle to understand why they supposedly require launching a thousand pods per second to use GPUs that need to fundamentally be installed across different baremetal machines. Once the GPUs are on different machines, if there are 1k+ such machines, just start putting them on different Kubernetes clusters. Build a scheduling layer above the Kubernetes control plane to decide which Kubernetes cluster to schedule the pod onto.

The whole thing stinks of, AI investors are throwing money at AI companies, so go to GCP and tell them to solve the problem at any price so that they can keep scaling without needing to build the scheduling layer above the Kubernetes control planes.

Because k8s gives you lots of other things out of the box like easy scaling of apps etc. Harder to do on VM:s where you would either have to dedicate one VM per app (might be a waste of resources) or you have to try and deploy and run multiple apps on multiple VM:s etc.

(For the record I’m not a k8s fanatic. Most of the time a regular VM is better. But a VM isn’t = a kubernetes cluster).

The reason to target k8s on cloud vms is that cloud VMs don't subdivide as easily or as cleanly. Managing them is a pain. K8s is an abstraction layer for that - Rather than building whole machine images for each product, you create lighter weight docker images (how light weight is a point of some contention), and you only have to install your logging, monitoring, and etc once.

Your advice about bigger machines is spot on - K8s biggest problem is how relatively heavyweight the kublet is, with memory requirements of roughly half a gig. On a modern 128g server node that's a reasonable overhead, for small companies running a few workloads on 16g nodes it's a cost of doing business, but if you're running 8 or 4g nodes, it looks pretty grim for your utilization.

> I see the appeal of K8s in dividing raw, stateful hardware to run multiple parallel workloads, but if you're dealing with stateless cloud VMs, why would you need K8S and its overhead when the VM hypervisor already gives you all that functionality?

I think you're not familiar with Kubernetes and what features it provides.

For example, kubernetes supports blue-green deployments and rollbacks, software-defined networks, DNS, node-specific purges and taints, etc. Those are not hypervisor features.

Also, VMs are the primitives of some cloud providers.

It sounds like you heard about how Borg/Kubernetes was used to simplify the task of putting together clusters with COTS hardware and you didn't bothered to learn more about Kubernetes.

because if you just do a few huge VMs you still have all the problems that k8s solves out of the box. Except now you have to solve them yourself, which will likely end up being a crappier less robust version of kubernetes.

In a large organization their more efficient to run on VMS. You can colocate services that fit together on one machine.

And in reality no one sizes their machines correctly. They always do some handwavey thing like we need 4 cores, but maybe well burst and maybe there will be an outage so lets double it. Now all that utilization can be watched and you can take advantage of over subscription.

VMs are a standardized system primitive. The “bare metal” bit with RBAC etc through the management layer / hypervisor.

K8s is pallets Vms are shipping containers

Systems / storage / network team can present a standardized set of primitives for any vm to consume that are more or less independent of the underlying bare metal.

Then the VMs can be live migrated when the inevitable hardware maintenance is needed (microcode patching , storage driver upgrades , etc etc etc). With no downtime for the vm itself

For what? Access to the control plane API?

The could team up with Microsoft, because their signup flow is fine but the login flow is badly broken.