Here's one I ran into recently: if a range has only 1 of 3 replicas online then it will stop accepting traffic for that range until it has 3 replicas again.
(for the folks at home, "range" is a technical term for 512 bit slice of the data - CRDB replicates at the range level)
So, in some code I wrote, I had account for not only 1) the whole DB being unavailable but also 2) just one replica being unavailable (they're different failure modes that say different things about the health of the system).
It's a good behavior! Good for durability. But I had to do some work to deal with it, spend an hour coming up with a solution, etc. There are databases that work at Twitter scale but no there are no silver bullets among those that do. You need full time engineers to manage the complexity and keep it online, or else it could cost the company shitloads of money - I've seen websites of similar scale where a two-hour outage cost them $20 million.
Like last time, and the time before that, and the time before that, and the time before that.
I understand the frustration with flavor of the week "best practices" and the constant churn of frameworks and ideas, but software engineering as a practice IS moving forward. The difficulty is separating the good ideas (CI/CD, for example) from the trends (TDD all the things all the time) ahead of time.
In fact it’s often meant to distract from the lack of progress.
It’s not the lack of progress that is the concern, it’s the subterfuge.