Hacker News Reader

Comment by mrkurt | Hacker News Reader

We chose Nomad and adopted Consul as a result. Nomad and Consul work well together.

I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot of sense for the things we need to do. And when I've had etcd blow up on me in the past, it's been similarly painful to recover from.

View on HN

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems.

Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

I would need a citation on the kubernetes control plane having any such hard limit. etcd is its own little snowflake, and I could very easily imagine it having some bad default value like that, or even kubeadm improperly configuring it

However, related to that, for big-time clusters (q.v. https://news.ycombinator.com/item?id=35174655 and https://news.ycombinator.com/item?id=25907312) one should without question move events over into their own etcd cluster: https://openai.com/research/scaling-kubernetes-to-2500-nodes...

It’s etcd default https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...

There’s also max object size of 1MB on the apiserserver side I believe

ByteDance replaces etcd with kubebrain [1], which is backed by their own KV store (TiKV seems also supported).

The single-group raft is the hard limit.

[1]: https://github.com/kubewharf/kubebrain

That's interesting, thanks for the link. I held out high hopes for pluggable KV in kubernetes for the longest time, but since that issue was closed WONTFIX I resigned my hopes

Heh, that kubebrain TODO is some "oh, really?"

* Guarantee consistence in critical cases

but I give them huge props for calling out Jepsen

Doesn't Consul have the similar storage limit btw?

I have seen very few strongly consistent distributed KV store that scales beyond 10GB+

It’s the underlying db limit (boltdb) which both etcd and consul use

I’ve run it with over 50G under heavy load (10k+ qps) and it was fine. It’s pretty sensitive to disk latency though

Raft is amazing and totally frustrating.

I think I understand how you're using it and curious if you've considered how AWS STS API manages their cross region syncing gets solved.

Thanks for the answer!

AFAIK doesn't Consul also use Raft?

Reply on news.ycombinator.com