The CAP theorem is so dumb and yet so influential. If you want to gain a more useful understanding of the trade offs of distributed systems, I highly highly recommend the “PACELC theorem” by Daniel Abadi:
PACELC is an improvement over CAP in some sense because it is a finer-grained characterization of distributed systems, but it still suffers from the problem that the stated properties are too strong. As Kleppmann explains in the article, there are systems that strictly only fulfill the partition tolerance part of the equation, yet are still very practically useful. This is because while they may not satisfy strict linearizability and availability, they have softer guarantees (e.g. defining "availability" as "99% of requests complete in 1 second") that we can rely on in practice. This reasoning extends to PACELC as well.
I am inclined to agree with Kleppmann. CAP, and by consequence PACELC are useful mental models in the extreme, but their characterizations are really strong. It seems to me that e.g. probabilistic guarantees based on actual failure statistics are much more useful for the real world, because they aren't as strong as CAP/PACELC guarantees.
Edit: I am also not sure if I agree with calling CAP "dumb". It's a very useful theoretical result in a particular model. I think the problem is when people try to extend the implications of the theorem beyond the restricted model it operates in.
In a different arena, I also know of no better book at the interface of distributed systems theory and practice than Kleppmann's. It's useful for engineers while being enriched by deep references to the literature. If that doesn't demonstrate mastery, I don't know what does.
The book chapters do an solid job laying the ground-work for those papers. The depth is in those references. Read them if you can!
The article claims that the CAP theorem only prohibits linearisability as a consistency model. The original CAP paper talks about "Atomic consistency" which is clearly at least linearisability. However as the proof formalizes, it's obvious that if you accept writes to totally split and separated systems which cannot communicate together (network partition), you cannot be consistent across the two parts. This is confusing because this kind of consistency only loosely matches the definitions in the ACID acronym which are geared around invalid database states in a single machine database, not about different requests to different database machines giving inconsistent answers.
Linearizability versus serialisability (and strict serialisability) are explained well by Peter Bailis http://www.bailis.org/blog/linearizability-versus-serializab...
I did not realize that consistency and atomicity also are overloaded between databases and distributed systems. That certainly adds to the confusion! I must have missed it when reading the original article.
More real-world experience is also really valuable, and an application-first approach to handling distributed systems works really well. I guess my hope is that eventually the theory becomes robust, matching the real world enough so that we won't have the problems of mis-applied theorems. IMO the best theories are intuitive and help us expose holes we could not catch in the real world.
If you can warranty high availability (by other means) then you can focus on consistency.
It also says that you should design your data so that becoming partitioned is not a problem.
What it doesn’t do, particularly, is help us build them.
Why?