As another example, Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust. Antithesis found some more bugs in HashiCorp's Raft implementation recently[0]. I'm sure etcd still has some kicking around.
Maybe this is a "don't throw the baby out with the bath water' problem but the general evolution of Go has been lackluster. I reach for Rust, Zig, and modern Java instead depending on the specific needs and constraints.
0 - https://antithesis.com/blog/2026/finding-bugs-in-raft-implem...
Are you saying that this implementation is wrong?
"This Raft library is stable and feature complete. As of 2016, it is the most widely used Raft library in production, serving tens of thousands clusters each day. It powers distributed systems such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, Hyperledger and more."
One of the most popular distributed DB is Cockroach which is written in go and also uses Raft: https://github.com/cockroachdb/cockroach/tree/master/pkg/raf...
You may be interested in knowing that the largest managed Kubernetes service in the world (AWS EKS) ripped out etcd for in favor of their homegrown consensus service for large scale EKS clusters: https://aws.amazon.com/blogs/containers/under-the-hood-amazo...
goBGP is arguably even worse.
I don't have a third place in mind that's even worth mentioning relative to these two.
(I have only a rather basic familiarity with go, but was considering gobgp for an infra project...)
[1] https://elegantnetwork.github.io/posts/comparing-open-source...
Gobgp is great if you want to embed it directly into a Go app though. Talos Linux has done that recently.
I've had to use it as a library sometimes, and it's also really poorly designed as a library. There's no consistent principle for what is exported and what remains internal, so almost anything non-trivial you do may require you to copy-paste parts of the library code because something happened to not be exported. So you must think, okay sure, but in return for that you get a minimal and stable API. Nope! It also has some of the most API churn of any Go module, already up to /v4 and that's only counting the semver-major breakage they bothered to acknowledge, not the many semver papercuts along the way.
0: https://etcd.io/blog/2025/autonomus_testing_with_antithesis/
Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable.
Its very much {reliable, performant, flexible} pick none.
Do these have reproducible test cases?
I've never had this be a problem in reality but AWS has a pretty good backbone. Maybe if you were running it over a pretty unreliable network this would be an issue?
That the world runs on Kubernetes is no qualitative statement about the correctness of its Raft implementation. You can say that it's clearly good enough to not matter most of the time, but that is a different statement. No matter who you look at, they're just cooking with gas like you do, and they can make mistakes in just the same way.
Now; I'm only attacking your argument. I do neither know nor particularly care about the correctness of that implementation itself. There's been better refutations of the claim you replied to in other answers anyway.
Thats basically it for starters, what non-technical solution do you propose?
But k8s was invented so that "dev" and "ops" teams could play nicely when they are different org structures, with their own budgets, deadlines, etc.
If it solves some technical issue then that's incidental.
Please, do describe your solution for the concerns above. So far Ive only seen equivalents of Greenspun's tenth rule
I didn't know Go just isn't a good language for it, but now that I know I'm no longer surprised at etcd being problematic.
Sure it may not be the best fit in a scenario where you want a cluster spanned over the entire globe (thats why GKE uses paxos-based Spanner instead of it) , but even spanned across an entire continent (in europe via glass fiber) it works quite well for me. Its one of the least problematic parts of the stack.
Could you elaborate?
> are you saying this implementation is wrong?
> That's not remotely what he's saying at all.
I'm v confused by this thread
That is literally what the comment says.
I hope my every competitor will take your advice to heart, as one of our competitors did when they read that "Go is not a memory safe language", so they wrote a blog about how they are porting to Rust. While our team was moving fast and using those "primitives that should almost never be used" around our long running production code base with success.
Some time has passed and now their company does not exist anymore and we have a lot of their clients.
Thank you!
> Go's internal data structures like interface values, slice headers, hash tables, and string headers are not immune to data races, so type and memory safety can be violated in multithreaded programs that modify shared instances of those types without synchronization.[113][114]
It's close enough for most purposes... but it isn't.
JS is fine for what and where it is, Rust is fine too. I just appreciate the stupid simple nature of Go and it does the job just fine.
If you want to see something that could potentially impact Raft's correctness, search the last couple of days of the HN front page for choreographic languages [1]. But none of these are even remotely mainstream enough to depend on for anything. Nor do I know if anyone in these languages has implemented Raft. A rather good test case for them, if any of them are looking. That's something that could actually help a Raft implementation's correctness, not just fiddle around the edges of local concurrency issues.
[1]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
The antithesis author states:
"we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft"> Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust.
That says that there are correct (i.e., bug-free) implementations in those languages. The GP noted
> "we’ve found bugs in every Raft implementation we’ve tested, ..."
which says that there aren't any correct ones. You then wrote
> I didn't say other languages don't have buggy Raft/Paxos implementations
which is a strawman. The issue is whether there are correct implementations. That there are buggy ones is irrelevant.
(FWIW I have no dog in this fight ... I'm just reading here.)
>> "we’ve found bugs in every Raft implementation we’ve tested, ..."
> which says that there aren't any correct ones
That only follows if the GP tested every Raft implementation in existence and no new ones were written since.
Edit:
> What are they implying by citing that? That Raft implementations in all languages have bugs?
That's what it says.
> I've already pointed out that is false.
You claimed that, and it's being disputed.
> Please let me know, since you're so comfortable speaking for them.
This has veered into bad faith ... I won't comment further.
> The antithesis author states:
> "we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft"
What are they implying by citing that? That every language has a Raft implementation with bugs? Yes that's probably accurate because lots of people make Raft implementations for fun and learning. Again, Go does not have a single Raft/Paxos implementation that is rock solid. I have seen many in C++, Java, and Rust that are doing tens of millions of requests per second in production for over a decade.
Is their point that Go is not the only language with this problem? My post already points out the track record is that Go is the problem for writing correct code in highly critical domains.
The only way this becomes useful for comparing languages is if somebody gives evidence of correct implementations in other languages. You're claiming they exist but with no evidence and suggesting they're secret. How do you know those don't have bugs? Did any concurrency bug experts do extensive testing on them? And can we disprove secret Go implementations of the same quality?
No, they are citing that every Raft implementation that Antithesis has tested has bugs. The etcd implementation you note in go that has bugs also does tens of millions of QPS and is over a decade old. How are you confident that the proprietary implementations that presumably haven't been fully tested don't have subtle bugs that don't show up in practice?
But good god, the second it gets to anything concurrency-related, it just loses its mind. As much as it's gotten vaguely ok to try to let the agents loose on some bits of the codebase, they simply can't even do table stakes stuff with the kinds of concurrency you see in real life.
Zig is also good at this but requires more up front design (thread-per-core, static allocation, etc.) and consistent checks to verify rules are followed.
It seems like claude code can code Rust pretty well with Opus, and I've started moving codebases away from Golang to Rust at work with Opus. Spin up an LLM and it cranks on it for a while, and as a benefit, I get easy apis to build on with other languages.
And that's the problem with Golang really, not that it's a bad language per se (all languages have footguns), but that the language interoperability story is terrible. Meanwhile Rust and Python/C/C++ go great together like peanut butter and chocolate. And I love it.
And deadlocks. "Fearless concurrency" helps a lot, but logic bugs are still possible.
If Rust's footguns are limited to getting past the borrow checker, and memory links with their reference counted memory structures, I'm all for it.
Once I've fought the language once I typically won't have to fight it again. This is my problem with C++.
Java leans heavily in the other direction: a lot of concurrency is added externally, without changing existing code, often in very declarative-flavored ways.
E.g. Future<T> serves as a foundation for a ridiculous amount of stuff, while Go forces channels for `select` whether they model your problem nicely or not, and they're very difficult (often impossible) to wrap without changing semantics.
There are very obviously lots of counter-examples for both langs (`synchronized`, rill in Go, etc), and I expect Go to become more Java-flavored in time (it already has moved this direction somewhat, and 1.27 will enable a lot more). But I think it's a fair summary of broad ecosystem habits.
1: https://tip.golang.org/doc/go1.27 (not yet released)
> while Go forces channels for `select` whether they model your problem nicely or not, and they're very difficult (often impossible) to wrap without changing semantics.
I understand that Go's concurrency model is based on CSP and fork-joins and the primitives like locks, but they are not mutually exclusive with concurrency containers, right? It's okay if the Go team's core philosophy is that channels are the universal abstraction, but I don't get why the community didn't produce 3rd-party containers as robust as JCTools.
Select, however, is encouraged very widely, and is used very widely, because it's very useful to be able to wait on any of multiple things efficiently. Select is great, you quickly grow to miss it in languages that don't have it. But select only works with channels, and often that means you're essentially forced to use channels where a mutex is much simpler and more natural. It also means tons of APIs are channel-centric because it's kinda the only non-blocking option (atomics exist, but that's a very different kind of non-blocking, and dramatically more error-prone for normal humans to write).
Once you go channel-centric for things like streams or concurrent RX style stuff, it does work, and some libraries do exactly that. But then you run into the historical limitations on generics, which makes for sometimes unnatural code, and channel performance is usually significantly worse than mutexes (it's still very fast, but you don't want to use it for extremely small things). And you are pretty much required to use those channels directly with select by hand because wrapping channels changes some semantics. And the semantics of those libraries are not mutex-y and are different from what most are already used to, because few other languages are channel-centric.
So you end up with high-level channel-oriented concurrency libraries that people only want to use for large expensive operations, thus are only designed for large expensive operations, which means it's not very common over all, and few develop the habit. E.g. there are quite a lot of goroutine-pool-helping libraries for ~seconds of work, but few functional-flavored generic parallelizing ones for opportunistic use.
It's part ecosystem, part language design, and part language history. You can do most of the Java stuff in Go with enough effort, but it just isn't done in practice very much, and it'll look and feel quite different.
the source you link to contradicts your own claims.
they say:
> we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft
(besides Go, that's 2 in Java and 1 in Rust)
Go obviously does not stop you from writing buggy code. Neither does rust or zig or whatever. Does go make it more likely to have bugs? Or a specific class of bug? Like, the real world is about trade offs.
Yes, it's a bit of blame the user which will likely get the retort of "but I thought Go was perfect for junior engineers?"
Yes, there are footguns but none of the points therein were compelling.
But Go is also perfectly good at single threaded polling loops.