Distributed Systems Classics (2017) | Hacker News Reader

Distributed Systems Classics (2017)

nvartolomei.com

351 pointsgrep_it75 comments

This is not a bad list for sure. Here are some deeper cuts for those looking for something a bit less mainstream:

"The Maintenance of Duplicate Databases" https://datatracker.ietf.org/doc/html/rfc677 (AFAIK the genesis of the use of logical clocks in distributed systems).

"Chain Replication for Supporting High Throughput and Availability" https://www.usenix.org/legacy/event/osdi04/tech/full_papers/... (Chain replication is how a huge percentage of real-world cloud-scale data replication is done).

"Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (The formalization of CAP, which caused a ton of very poor trade-off thinking in the decade that followed by defining Availability in a very goofy way. Still a classic.)

"Paxos Made Live" https://research.google/pubs/paxos-made-live-an-engineering-... (Brought a much-needed engineering perspective to a conversation that was largely theoretical up until this time.)

"Practical Byzantine fault tolerance" (Moved the conversation on Byzantine faults forward significantly).

This is just a short selection. There's so much good stuff going back in the 70s and 80s distributed database literature, for example (and in the modern systems and DB literature too).

Lamport lists his collected works on his site[0] along with some commentary for each one, and directly mentions, The Maintenance of Duplicate Databases, as the origin for logical clock paper. Many of the notes have interesting commentary.

> Many computer scientists claim to have read it. But I have rarely encountered anyone who was aware that the paper said anything about state machines. People seem to think that it is about either the causality relation on events in a distributed system, or the distributed mutual exclusion problem. People have insisted that there is nothing about state machines in the paper. I've even had to go back and reread it to convince myself that I really did remember what I had written.[1]

0: https://lamport.azurewebsites.net 1: https://lamport.azurewebsites.net/pubs/pubs.html#time-clocks

Question from a recent CS grad here, who just learnt about CAP: what do you mean by “very poor trade-off thinking in the decade that followed”?

A Critique of the CAP theorem by Martin Kleppmann.

https://arxiv.org/abs/1509.05393

Basically naive CAP theorem assumes "no latency (delay) allowed", in which case, yeah, sure, the quorum must agree or you are writing to a split-brain quorum somehow. But what if delay sensitivity is a knob that can be tuned, and writes have conflict resolution?

Martin Kleppmann's book Designing Data-Intensive Applications is considered a useful tome for anyone who needs a insight in the complexity involved in distributed systems (hint: getting data in ram flushed to an SSD also involves multiple moving parts)

Basically, many people convinced themselves that the formalization of CAP means that systems need to choose between highly available and strongly consistent, and hence chose eventual consistency. This is, partially, because Gilbert and Lynch define "availability" to mean "available to all clients, even those on a minority side of a partition". The much more useful "available to a majority of clients" is achievable at the same time as strong consistency in presence of a single partition, and this is the common cloud failure mode.

See https://brooker.co.za/blog/2024/07/25/cap-again.html for a longer take.

Other, much more reasonable trade-offs, lead to eventual consistency too. Mostly latency optimizations, but many of those also lead to non-zero RPO and so are undesirable for multiple reasons. We discuss some of this in section 8 of the DSQL paper: https://arxiv.org/pdf/2607.13276

Thank you both! That was really enlightening & made for interesting reading!

Though the note at the bottom of the article to teachers to not start with CAP has me concerned about the quality of other content I’ve seen thus far …

> Chain replication is how a huge percentage of real-world cloud-scale data replication is done

Is this still true today? I know that Facebook does [1], but I don't have references for any other places which use it.

[1] https://engineering.fb.com/2022/05/04/data-infrastructure/de...

Inside multiple AWS products (including DynamoDB, Kinesis, and Aurora DSQL) is a system called Journal that moves a ton of data. It uses a variant of chain replication.

EBS is also a chain replication variant at heart, and moves even more data.

I am a Lamport admirer.

I gradually realized that Lamport is more of the godfather of distributed system than Hinton is to deep learning. Lamport is less prominent than Shannon is to information theory. Shannon is the closest to any title of "gold-like" figure to a scientific discipline of universal relevance in modern society.

Lamport specifically revealed a philosophical connections between computer systems and physics, in the parallel of distributed consensus to relativity theory. To me, the enlightenment is that, the relations between events happening in a distributed system, is more fundamental than their absolute ordering, thus the central role of an "observer". I haven't really analyzed if this realization is from Lamport's papers, or my general reading and thinking, but I am moderately confident that general readings are heavily influenced by Lamport's papers, or can be traced back to be compatible with Lamport's thinking. I have not seriously study if this connection is valid in depth, which might be another pure amateur speculation of mine.

One thing I think Lamport falls short is that his writing is not easy to read and understand. I unconciously feel that Lamport (and Dario from Anthropic) probably share a hidden sense of intellectual supriority grew from their own experience throughout their career. So their writing (and Dario's gospel) all share a unchangable sense of narration from their own delicate and graceful ideas, much less of faciliating the understanding to their audience. In this cateogry, Shannon is abosolutely superior in any measure, in his writing, ideas are so naturally presented, although the implications of the ideas remain elusive due to the inherent depth.

Also, among the 3 prominent figures of modern AI: Hinton/Bengio are more like Shannon, Lecun is closer to Lamport.

Enough random rambling. Lamport, as indicated by the outweight presence in this list, is no doubt the single most important scientist in distributed systems.

> the relations between events happening in a distributed system, is more fundamental than their absolute ordering

The important thing in most distributed systems is having an order. Having a single observer serialize events as it receives them is so much more tractable than trying to use absolute order. Using absolute order requires very precise time synchronization which is hard; using absolute order requires knowing when you have received all the reports of events that already happened which is hard. Determining a designated observer isn't typically easy, but having it determine the order it observes events is easy. If two events happen at a similar time it's typically not a big deal which one is considered first as long as all nodes will agree on the result --- let the designated observer just pick the first one it sees works pretty well. If your report takes an unexpectedly long time to make it to the designated observer, then it won't be first and you'll deal.

Much better than trying to figure out unknowable questions of relativity. :P

> absolute order requires very precise time synchronization which is hard

Presumably relativity is the reason precise time synchronization (and thus absolute ordering) is hard.

That forms a basis for the difficulty. But then there's additional layers of difficulty in real networks where the path between nodes is often asymmetric, and you may also observe that elapsed time (A -> B -> A) is sometimes greater than elapsed time (A -> C -> B -> C -> A) or (A -> C -> B -> A) or (A -> B -> C -> A)

Doesn't each node just need to know its exact path (or latency of same) to the time source each is synchronizing from? The path from node to node doesn't matter because we're ordering log entries from the timestamp of the receiving node.

Imagine you take three atomic clocks, synchronize them, and move each within exactly 1 meter of one of those nodes, directly connected in an identical manner. Relativistic effects are constant. If there are elevation differences, you factor it once and done. The problem with this setup is not relativity. It is quantum uncertainty (and various interference sources, thermal radiation, etc).

How do you know the latency of the path if you haven't synchronized the clocks...

You might be able to control the latency on a LAN, but once you have servers in different locations, good luck. (GPS helps a lot, of course...)

There is this command called 'ping' that tells you that. But in my example, I included the network topology as a known, static factor.

> There is this command called 'ping' that tells you that.

Ping tells you the round trip latency. As I mentioned, asymmetric routing is common and very difficult to measure.

> But in my example, I included the network topology as a known, static factor.

Ok, so spherical cows.

yes, it can be easier to have a central serializer for events, but that certainly makes things problematic for fault tolerance and basically excludes large-scale solutions. using an agreed-upon post-hoc ordering based on timestamps is certainly another way, but really only if you're working in a paradigm that lets you impose it (like mvcc) or doesn't care (like crdt).

personally I find when you view events in a distributed system as a partial order, its more liberating than confusing. its not unusual to assume that there is some kind of canonical event ordering that we have to preserve, when its often just not semantically important. so its a useless constraint that can impose complexity and limit the solution space. the partial ordering exposes the real causal constraints.

There is no such thing as a general order; if people accept this then the problem relaxes and we stop trying to design impossible things.

Nodes shouldn't be observers. End users should be the observers. The ordering of event only matters from the perspective of the "I'm adding this to my shopping cart and clicking the buy button and then it said I bought it" story being internally consistent. The observer here is the end user and the end user knows what happens-before relationships were.

Databases should flip this ordering inside out and let the observers (or proxies for them) report what they did in which order. And it just happens that protocols for doing this already exist: distributed tracing encodes these relations.

A meaningful global order doesn't exist and attempts to conjure one only conjure a meaningless one, or one that throws away a lot of turns trying to perfectly order everything from some arbitrary internal observer.

> "I'm adding this to my shopping cart and clicking the buy button and then it said I bought it"

If there's only one of the item to buy and two users wanted to buy it at the same time, it can only be shipped to one of them.

From each user's perspective, their add to cart (and maybe their click to buy) happened before the other user's, but alas, there is only one item available for immediate shipment.

Allowing customers to arbitrate a dispute over control of the item doesn't always go so well; see examples on Black Friday.

Even without a conflict over a mutually exclusive resource, it is valuable in many contexts for all users to share the order of observed events. That's difficult if user agents send events directly to all other agents and report observations to the user as soon as the remote reports arrive.

If A and B both report events and some users see A then B while others see B then A, there may be confusion and a poor user experience. Of course, for some things, relative ordering is unimportant.

Hot take of the day:

Computer scientists are in denial about it, but CS is a branch of theoretical physics, not mathematics. You can point to this or that model of computation, such as lambda calculus or mu-recursive functions and try to claim its abstracted well beyond the particular laws of physics for some specific universe, but they all have some kind of rate limit built into them... and where does the motivation for this idea, that it takes something (time, space, work) to compute something ultimately come from? That's right - from underlying physics itself[1] - from the Bekenstein bound or Bremermann's limit or the like.

Even apparently non-physically-realizable models of computation like non-deterministic Turing machines are ultimately informed by and motivated by concepts in physics... otherwise they would just be examples of chmess[2] and of no interest to anyone. Computer science is of course somewhat abstracted from the details, but no more so than, say, thermodynamics, where concepts like entropy or Gibbs free energy can be studied in the abstract without reference to whether we are talking about a gas of non-interacting molecules or the spins of a bunch of electrons trapped in a lattice.

So, it's of no surprise whatsoever that the fundamental problems of distributed computing are ultimately the same as those found in the relativity of simultaneity[3]. You've all been studying the same things all along, just with different tools and at different levels of abstraction.

[1]: https://en.wikipedia.org/wiki/Limits_of_computation

[2]: https://link.springer.com/article/10.1007/s11245-006-0005-2

[3]: https://en.wikipedia.org/wiki/Relativity_of_simultaneity

I suspect this is old news for you, but just in case you haven't heard of it, check out Feynman's Lectures on Computation. A surprising amount of the book is still relevant, and it's fun how much he always brings everything back to the physics.

Related hot take

Maybe thats why one of the not-so-bad ideas how to go about distributed systems came from a guy who was trained as a physicist and used to complain to his fellow programmers that “a lot of systems actually break the laws of physics”[1]?

> In distributed systems there is no real shared state (imagine one machine in the USA another in Sweden) where is the shared state? In the middle of the Atlantic? - shared state breaks laws of physics. State changes are propagated at the speed of light - we always know how things were at a remote site not how they are now. What we know is what they last told us. If you make a software abstraction that ignores this fact you’ll be in trouble.[2]

[1]: “The Mess We’re In”, 2014 https://www.youtube.com/watch?v=lKXe3HUG2l4 [2]: https://news.ycombinator.com/item?id=19708900

And yet we have CRDTs.

CRDTs are eventually consistent data structures

A thought provoking leading question I have is (1) a space heater throws power into a coil and generates heat and nothing more in the world (2) an LLM into a GPU throws power into the GPU and generates heat and organized information.

Is (2) generating less heat so that the laws of thermodynamics account for the increased amount of "useful stuff" generated (the organized information)? Switch (2) for a bitcoin miner if you prefer.

Is the act of increasing order in the data, somehow costing some energy? Is there a fundamental minimum cost to flipping bits? Something about the switch of energy to matter and back, seems involved here.

I think the nature of computation is holding some secrets about the meaning of the universe, if there is any.

> Is there a fundamental minimum cost to flipping bits?

Yes:

https://en.wikipedia.org/wiki/Landauer%27s_principle

But modern computers are nowhere near this theoretical limit, nor any of the other limits I mentioned above. Nevertheless, most of heat generated from modern CPUs does come from bits turning on and off. Each transistor is a tiny capacitor, that holds a charge when its ON. When it switches OFF, it dumps that charge down the drain, creating waste heat. This is a limitation of our technolgy, not a fundamental limit of physics.

Could be worse, though; early chips would disipate heat even when they weren't doing anything. CMOS improved this enormously by pairing up "complementary" transistors so current only flows when something changes.

Still, from the universe's point of view, what we consider a super advanced computer is a lot closer to a space heater than anything that pushes up against its computational limits. Consider, for example, that quarks operate on time scales of 10^23 Hz, and the universe is happy to run three of those in every proton in every star in the universe. In fact, it runs 10^24 of them for one CPU, and that same CPU can't even simulate the quarks of one proton in real time.

Let's face it: we're like kids in Minecraft who think it's cool watch a calculation of 2+2 trickle through a redstone computer in a minute, while the GPU is rendering a billion triangles every second to give them that view.

I think you should consider the gigawatts of heat generated to create the LLM model to make it a more fair comparison.

> One thing I think Lamport falls short is that his writing is not easy to read and understand

Interesting; I actually grew to be a fellow admirer of Lamport primarily because I actually found his papers to be a lot more approachable and relatively straightforward.

Some other classics (applied dist systems):

- DDB: https://www.allthingsdistributed.com/files/amazon-dynamo-sos...

- MapReduce: https://www.cs.princeton.edu/courses/archive/fall13/cos518/p...

- Spark/RDDs: https://www.usenix.org/system/files/conference/nsdi12/nsdi12...

- BigTable: https://static.googleusercontent.com/media/research.google.c...

Worth mentioning that Dynamo (the classic 2007 paper) and DynamoDB (the modern AWS product) have fairly little in common architecturally.

For DynamoDB, check out https://www.usenix.org/conference/atc22/presentation/elhemal... https://www.usenix.org/conference/atc23/presentation/idziore... and my analysis of the differences here https://brooker.co.za/blog/2025/08/15/dynamo-dynamodb-dsql.h...

Nice distinction, thanks!

As always, lists like these don’t include Joe Armstrong's PhD thesis “Making reliable distributed systems in the presence of software errors” - http://erlang.org/download/armstrong_thesis_2003.pdf

Are there any other works that should also be included that you know of?

Not OP, but a good resource for distributed consensus specifically is Tim Roughgarden's YouTube playlist "Foundations of Blockchains" [0] (hear me out, despite the title - see below). It's 85 videos over 12 lectures, rigorous, very well explained (but assumes some CS fundamentals). Not original work, but gives the conceptual framework and background so that one can read these classic papers in context.

Per-lecture reading lists are on the course page [1]. He also points at Elaine Shi's Foundations of Distributed Consensus and Blockchains [2] and Andrew Lewis-Pye's Consensus in 50 pages [2] as background, though these are more textbooks, not papers.

Remarkably, the first 7 lectures deal with permissioned systems, recapitulating the classic results of consensus in distributed systems (Dolev Strong, FLP impossibility, CAP) - no blockchain in sight. This takes us to the state of the art at the end of the 1990's (with algorithms that can achieve consensus in the presence of byzantine failures, namely Byzantine Paxos and PBFT, though he discusses a modern variant, permissioned Tendermint from 2014).

Lecture 8 stays permissioned and proves consistency and chain quality for the longest-chain rule. Only at Lecture 9, with proof of work, does anything specifically blockchain appear; then L10 block rewards and selfish mining, L11 transaction fee mechanism design; L12 proof-of-stake sybil resistance with the whole litany of attacks possible there.

Very good series in my view, and shows how little technical merit this whole blockchain circus has - nearly all the great properties people tout (reliability, consistency, audibility, availability) can be achieved with good old permissioned tech more efficiently, with pretty instant and deterministic finality.

Anyway, I found the series worth watching for the classical consensus material alone.

[0] https://www.youtube.com/playlist?list=PLEGCF-WLh2RLOHv_xUGLq...

[1] https://timroughgarden.github.io/fob21/

[2] https://elaineshi.com/docs/blockchain-book.pdf

[3] https://lewis-pye.com/2022/08/15/consensus-in-50-pages/

No disrespect meant to Mr. Armstrong, but it's possible it's never listed because it's basically a textbook. It's 295 pages. The rough average of all papers in OP is like 15 pages.

His thesis is only slightly longer than typical PhD theses, I think.

Yes, but the other things in the OP list aren't theses, they're journal papers.

I would say 300 is more than slightly longer. Varies by institution and style, but 100k words, 4-5 collated papers, 100-150 pages maximum are quite common.

I would be concerned as an examiner if this came across my desk, more so if I read the colophon where the author comments that they intended to write their own typesetting system, before reading Knuth and wisely concluding that they were unlikely to do anything better than TeX. Top tier yak shaving there.

One can skip the Erlang-specific things/the description of the programming language.

Just the first 2 chapters (~30p) + the Conclusion (~10p) contain a lot of useful food for thought.

Does anyone use Elixir/Erlang anymore?

WhatsApp backend in Erlang + used in telecoms

The "WhatsApp used Erlang" story was a decade ago, doubt it is true now.

There's still a lot of WhatsApp Erlang tooling being published, so I'm pretty sure it's mostly still Erlang. I think I still know a couple people who would share with me if it was rewritten, although I left in 2019.

Erlang is simply the right language to build a chat server with, and it doesn't make sense to rewrite it in another language, even though Facebook did that with the server for Messenger. :P

Certainly parts of the WhatsApp stack have changed considerably. No more FreeBSD / no more only a handful of OS processes per node, much less Mnesia (not sure if there's any Mnesia left... I don't get that level of information). I've seen signs that they may have returned to large nodes, so they might be back to millions of connections per node, but that was very much not the case when I left; connection counts had dropped on our FreeBSD nodes because we were doing a lot more on the chat channel and then again because the Facebook nodes we migrated to were so much smaller than our FreeBSD nodes.

I doubt they changed out Erlang/Elixir, as it is mature backbone infrastructure. It has run many telecoms for over 30 years without issue.

The problem is finding talent that knows both distributed systems, and Erlang/Elixir. But Erlang/OTP can replace a lot of expensive kludges. =3

> But Erlang/OTP can replace a lot of expensive kludges.

If you can convince your employer of that. I'm working on a system today (and not for the first time) that could have significantly reduced complexity by switching to Erlang (or Elixir) and OTP. Amusingly, it's a glorified, in-house telecom system that was developed because they chose not to buy an existing solution.

Ah, I just recommended simple AMQP (packed BSON <128k messages) for RabbitMQ with client x509 cert GUID access as secure "middleware" to link all the legacy systems together. The addition of a strict sanitization producer/consumer process bound to each cpu core handles a sliding window idempotent operation handler. Or, people can corrupt their own data if they really try for a finite period, but the system eventually becomes consistent.

Notably, it is possible to rebuild or move the entire infrastructure by issuing a signed resend-logs-to-new-consumer request to clients or servers.

If you start with a dumpster fire at a dried fish factory, than 3 week solutions that last 6+ years under load look pretty nice in retrospect. Your boss is probably contemplating why most Erlang/Elixir people don't work for clown wages. =3

>> But Erlang/OTP can replace a lot of expensive kludges.

> If you can convince your employer of that.

It's probably easier to find an employer that's already convinced. But that's not easy either. :P I'm working in Rust now, because that's what the employer wants to do and it works enough. Half of my work problems would evaporate if we switched to BEAM, but I'd still have the other half and new ones and I'm not sure the rest of the team would be as excited as I am. And they almost certainly wouldn't let me hot load code, which means less fun and less fast iteration :(

Convincing an employer to switch languages is generally hard, regardless of which language, but switching to a system built on BEAM is probably a bigger change than most.

Ok I will bite, what do you think WhatsApp backend servers run on? Python, NodeJS?

Yes, but it is boring telecom projects... so most people aren't interested.

Scala is kind of a more modern alternative.

If doing a twitter like platform, elixir with Phoenix channels could handle around >20k users per host. Very few other options work for that use case. =3

goroutines?

How the OTP handles Erlang/Elixir parallelism is kind of interesting. =3

https://blog.stenmans.org/theBeamBook/#_concurrency_parallel...

reliable? we all wish...

In distributed systems the fist lesson is resilience is more important.

Spend enough time in a computer lab, and you will see things halt and catch fire on occasion. Especially if it has a bunch of GPUs pinning the utilization 24/7, or a cheap power supply in the cluster. =3

Here is my list: https://muratbuffalo.blogspot.com/2021/02/foundational-distr...

Lamport is the author of more than half of these, it's crazy how influential he is in this space

In addition to making LaTex which has almost nothing to do with it

It's the thing that baffles me the most with those people, the Knuths and Lamports and Carmarcks, and whatnot.

It's not like they have light schedules - they do research, they have classes, they attendconference, and they have jobs, etc...

And some benefit from tenure, which give them time to deeply focus on some topics.

But I can't imagine how I would write tex in ten years in Pascal on archaic machines if I had all the time to myself.

Doing that in the middle of understanding more computer science than my eyes would ever read ?

What do they do with their time ? What do they don't do ? Are they able to skip the eating and pooping and sleeping and cleaning their house and picking up parcels at the post office and grocery shopping parts ?

Or a proto Unix being written in three weeks. Concretely, what are the days of those three weeks made of ? Especially since it's famously when the spouse was NOT there to handle the housekeeping...

I see not amount of "life hacks" and "atomic habits" and whatnot helping with that. But it's probably just as natural to them as biking - and you can't explain biking, I guess ?

A favorite Knuth story to illustrate: He was re-writing the 8080 code for the processor that ran our new Alphatype CRS Phototypesetter, since the manufacturer's code wasn't suitable. Real-time, 7 levels of simultaneous interrupt stuff, since it controlled two stepper motors that moved the lens around the photographic paper, plus four custom font outline-tracing S100 cards that flashed vertical segments of characters at just the right moment on the CRT that the lens then focused onto the paper. No debugability other than a 4x4 keyboard and a 15(?) character display. I was responsible for the code on the other end of the RS232 port that connected it to the DEC mainframe, sending down character outlines on the fly to the very limited memory on the 8080, along with the properly-sorted character placement info ultimately from TeX.

Anyway, we'd been doing morning, noon, and night debugging sessions together for a number of days. On maybe the 3rd morning, I arrived at his office and was speaking with his secretary of many years, Phyllis Winkler, when Knuth came in. He handed her a stack of maybe a dozen or two pages, saying "Here's a paper I wrote; please type it in." When? When did he write it? It's still a mystery to me.

My brother was something of a prodigy. Whatever it is that people like Carmack have, he had it too (and Carmack was one of his personal heros).

I would say with my brother it wasn't just that he was analytically extremely sharp, it's that he could focus it on things in a way that bordered on unhealthy obcession.

For example during undergrad he took a semester off because he burned himself out taking an insane course load, while also working on research as an undergrad.

Anyhow he spent that semester and the following summer at my parents house, and he spent months in basically the same pattern. Work on his research topic on the family PC until he needs to read a paper or think about something, at which point he'd roll the office chair over to the nearby couch and flop onto that. Every once in a while he'd disappear to his old bedroom to sleep for a few hours or take a shower or such.

And I mean he lived basically half a year like that, just relentlessly drilling into his research topic with every waking moment.

I'm assuming this was a while back, mind sharing how far he went or what he was able to produce with this focus?

He was planning to go into academia, but ended up abandoning his PhD ABD because he got exposed to the negative interpersonal politics that can happen in academia and realized that was not for him.

So he went to go work a dozen years at a hedge fund and then retire to be a stay at home dad.

His primary research interest was logical frameworks, constructive mathematics, etc. His PhD topic was on using linear logic to verify concurrent programs.

Good for him! That's the dream!

I'll add my stories to the 2 brother comments, in my prepa my best friend was by far the best in class. He did the 2 year program in chemistry in the first 2 months by himself. He arrived once in exams after a full night of re-reading the 3 LOTR books and still nailed the best grade effortlessly. And to his own admission, he was far from the most productive / smarter guys he knew.

I remember listening to him and a teacher, the teacher would give some anecdotes on what he's seen at the ENS. People reciting world chess championship games at night to fall asleep. Reading massive math books (1000+ pages) in an afternoon. Another story I heard in a different context from another teacher, math students were assigned a python project, after an introductory course. They never did any programming before. One student wrote 300 LoC in a single function and showed it to the teacher. It didn't have a main function, so the teacher asked him how he tested it. The student didn't know what he was talking about. They added what's needed to run the function and it worked.

So, it still begs the question... "How ?" Or, more precisely: "How the f... ?"

Clearly this goes beyond "Grit" [1], here, right ? (Those people do not seem like they have to coerce themselves into focusing that much.)

Is there a known (somewhat rigorous or not) study of those "super geniuses" ? I know that the "memory champions" are covered in "Moonwalking with Einstein" [2] ; but that seems like a different beast...

[1] https://www.simonandschuster.com/books/Grit/Angela-Duckworth... [2] https://www.goodreads.com/book/show/6346975-moonwalking-with...

Ah, I wouldn't know of any actionnable way to unlock this for yourself unfortunately. I think those people are gifted. There is definitely research (and documentaries around that research), you can look for Julian Stanley "Quick learners" or Maryam Mirzakhani and there are of course plenty of it. But again, it's not actionnable to watch a 13yo solve a very hard problem in 4 different ways in 20 minutes.

Unfortunately I think it is as much a curse as it is a gift. One of my parents' friend is a math researcher. He is not at the absolute end of the spectrum, but he is the kind of people who will write on the bathroom walls because he has an idea NOW and it cannot wait a minute to explore. He forces himself to do competitive cycling and can't enjoy more casual hobbies because if he does not spend 100% of his energy and focus to something, his brain will just do math.

Some of these probably aren't exactly classics but I do love:

- rendezvous hashing / consistent hashing

- Hybrid logical clocks

- "Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS"

- "Scaling Replicated State Machines with Compartmentalization" (much more recent but really one of my favorite papers)

Rendezvous hashing is so simple and elegant.

I implemented a library for rendezvous hashing with skeleton in order to get a similar performance as consistent hashing; it gets much more complicated and IMO loses some of the elegance once you add the skeleton. Most of the challenge is that the skeleton structure relies on a tree and so has to be rebalanced sometimes and this means that sometimes you have to move around more keys than ideal. The startegy you can use to perform the rebalancing can become very complicated.

Though the 'virtual nodes' in consistent hashing also add a lot of complexity there.

I haven't updated this in quite a while, so probably more than a few broken links: my list of distributed systems reading lists.

https://gist.github.com/macintux/6227368

Obligatory shout out to Morris and Kaashoek's class at MIT - they update the reading list every year, and it's always great: https://pdos.csail.mit.edu/6.824/

I am going through this right now. I finished Lab 3. I also made my own notes about some of the papers they discuss, specifically Spanner.

https://sudk1896.github.io/2026/09/03/Spanner-notes.html

Highly recommend this course.

I found myself fascinated by two topics -- Synchronization and Virtual File System in kernel land. Do they have similar websites? I know SunOS was the first one who implemented VFS and they wrote a white paper about it back in the 90s.

I want to work on coding for a service large enough to involve distributed systems someday. I keep studying the theory, but still...

Try doing the assignments from MIT 6.5840 Distributed Systems course by Morris and Kaashoek.

Thank you. Since most of my work involves small to medium-scale projects, I really appreciate the advice. I will definitely look into it.

Reading this brings back memories of wrestling with eventual consistency. So much has evolved, yet fundamentals remain.

> Satoshi Nakamoto. 2008. Bitcoin: A Peer-to-Peer Electronic Cash System.

The greatest anonymous dude alive.

18 years later the odds of him being dead has risen significantly.

Reply on news.ycombinator.com