Cloudflare Meerkat - Globally distributed consensus
blog.cloudflare.com
blog.cloudflare.com
I think the unique idea here is supposed to be QuePaxa's idea of avoiding timeouts for ensuring liveness. The actual discussion of QuePaxa is limited to one paragraph at the end, and tbh only a couple sentences of that paragraph.
I feel like the article could've been titled "Consensus protocols and linearizability: a brief explainer", or "Paxos vs Raft", or similar. It just doesn't feel like it communicates what it claims to communicate, and is a bit confused on who its audience is, just IMO.
Which would also be a good read, but this article also isn't that. It doesn't discuss their experience deploying the protocol, aside from the following statement:
> Meerkat is not deployed to production, but we have run multiple proofs-of-concept with up to 50 replicas distributed around the world, to great success. Leaders in our proof-of-concept clusters constantly fail, and the cluster keeps operating with no increase in error-rate.
I think it would've been more interesting to read why Cloudflare chose the specific algorithm they did, see an example of a pathological but common situation Cloudflare sees at their scale that makes other protocols unsuitable for them, therefore they made X choice and this led to Y gains in production (or on dummy workloads, or whatever). As it stands, there's nothing here actually specific to Cloudflare's workload or deployment. It doesn't even state their use-case beyond "small pieces of control plane state (e.g., leadership for replicated databases)"
Maybe the blog post is just premature. It would be much more valuable if they posted it after actually having run it in production and validated the strengths and weaknesses with real world data.
They could have picked something far more concrete (for Cloudflare's business and the type of services they deploy) than "AI model" there, which is clearly picked just to make it seem 'current' and on-trend.
But this includes READ operations too! You have to get global consensus for every read! Most distributed systems have additional complexity and latency on write paths so that reads can be completely local. If you can accept slow read operations this seems like a great trade off, but I think that is going to relegate this to niece usage.
Of course, you could build database that uses meerkat to do leader election, and then neither reads nor writes need to go through the consensus algo. But then you've basically just reproduced conventional distributed db, and you're back to the timeout problems this was supposed to solve.
E.g. 1. Intro about issues with concensus 2. Intro to QuePaxa 3. Comparison to other algos that are close to it 4. Mentioning active work on implementation via Meerkat and intent to bring to production with followup posts.
As always when it comes to concensus it's all about trade-offs. And with QuePaxa that might be the increase in messages (note: I don't mean message round-trips). We'll see how it goes but it will definitely be interesting.
Take aways:
* it’s not in prod yet. I suspect those many round trips are going to get expensive on median aka typical redistributed deployments. Curious to see how it goes once in the wild.
* they say it isn’t likely suitable for eg databases.
* they talk about formal verification, which is good and feels appropriate.
Looking forward to seeing more!
Who else than Cloudflare (or similar company in expertise and size) would be a better fit to implement distributed consensus?
This is obviously true, and there are surely more web devs than, say, compiler engineers.
But then the logic seems to go as far as to imply nobody does or should be doing these things. Where do they think the “off the shelf” solutions came from?
I worked at <big internet tech company>, and many critical systems written in the 90’s and early 2000s used bespoke compilers, caches, and consensus protocols that were hand rolled by the original developers.
Back then, most of these ideas were already present in academia. But the industry tech wasn’t quite there yet to run these huge services at scale without running into problems (paying a vendor, licensing, the open source solution not quite there or tunable enough to easily integrate it at the needed scale, no company wide consensus on a shared approach yet).
A lot of these handwritten implementations eventually informed the “off the shelf industry solutions” we have today.
I think that’s what gets me down about tech these days. There’s a weird almost anti innovation attitude, which is the opposite of what made me fall in love with this profession. I’m not saying to hand roll a consensus algorithm at your next start up. But there’s definitely a vibe these days that any sort of theoretical, creative, or innovative thinking is suspect. Get back to selling ads!
Yeah they kinda suck and make mistakes right now, but tech only gets better.
It’s no different than 3D printing. Instead of cutting and joining a bunch of stock parts, you just print the desired part.
In the same way printing with plastic kinda sucks, the tech will improve. Now you can print in metal or in resin without supports.
The future will be take all the ideas from various off the shelf products but tightly integrate them together for my specific use case
1. I'd bet they either already do log-routed config, or simulation testing finds this race before production does.
2. The first part of Meerkat that will need clocks for correctness will probably be stale local reads, not the core consensus protocol.
This is incorrect.
AWS, Azure, GCP still use etcd in their customer kubernetes services.
"EKS Ultra Scale" uses an in-house Journal
https://docs.cloud.google.com/kubernetes-engine/docs/concept...
https://cloud.google.com/blog/products/containers-kubernetes...
Interesting
The real threshold is multi-region writes on a hard latency budget, and even then a single-region primary plus accepting cross-region read latency beats eating a consensus round-trip on every write for a lot of teams. Curious what workload pushed you past single-primary - usually a better story than the impl itself.
1. This article is about Cloudflare developing for their own internal control plane--I'm fairly certain their requirements preclude a single Postgres
2. Even if you're not Cloudflare, you may not want to be limited by the availability of a single Postgres instance.
True, but the article claims both strong consistency (sort of) and linearizability; that is basically a serializable writer and a lock; if I read correctly (I haven't read all the details), the article conflates both; afaik it is possible to have linearizability with eventual consistency; algorithms to solve concurrent writes and having multiple writers in raft is a "solved problem". sad to see such a spotty article on such important topic by a major company, even if the underlying paper is strong. This feels like manure.