Hacker News Reader

Comment by aeyes | Hacker News Reader

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems.

Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

View on HN

I would need a citation on the kubernetes control plane having any such hard limit. etcd is its own little snowflake, and I could very easily imagine it having some bad default value like that, or even kubeadm improperly configuring it

However, related to that, for big-time clusters (q.v. https://news.ycombinator.com/item?id=35174655 and https://news.ycombinator.com/item?id=25907312) one should without question move events over into their own etcd cluster: https://openai.com/research/scaling-kubernetes-to-2500-nodes...

It’s etcd default https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...

There’s also max object size of 1MB on the apiserserver side I believe

ByteDance replaces etcd with kubebrain [1], which is backed by their own KV store (TiKV seems also supported).

The single-group raft is the hard limit.

[1]: https://github.com/kubewharf/kubebrain

That's interesting, thanks for the link. I held out high hopes for pluggable KV in kubernetes for the longest time, but since that issue was closed WONTFIX I resigned my hopes

Heh, that kubebrain TODO is some "oh, really?"

* Guarantee consistence in critical cases

but I give them huge props for calling out Jepsen

Doesn't Consul have the similar storage limit btw?

I have seen very few strongly consistent distributed KV store that scales beyond 10GB+

It’s the underlying db limit (boltdb) which both etcd and consul use

I’ve run it with over 50G under heavy load (10k+ qps) and it was fine. It’s pretty sensitive to disk latency though

Reply on news.ycombinator.com