Distributed Databases Should Work More Like CDNs
cockroachlabs.com
cockroachlabs.com
RethinkDB was Master-Slave (strongly consistent) with an amazing developer community and actually survived Aphyr's tests better than most systems.
CockroachDB is also in the Master-Slave (strongly consistent) camp, but more enterprise focused, and therefore will probably not fail, however their Aphyr report worked... but was unfortunately very slow. But hey, being strongly consistent is hard and correctness is a necessary tradeoff from performance.
Other systems, like Cassandra, us (https://github.com/amark/gun), Couch/Pouch, etc. are all in the Master-Master camp, and thus AP not CP. Our argument is that while P holds, realtime sync with Strong Eventual Consistency is good enough yet has all the performance benefits, for everything except for banking. Sure, use Rethink/Cockroach for banking (heck, better yet, Postgres!), but for fun-and-games you can do banking on top of AP systems if you use a CRDT or blockchain (although that kills performance, CRDTs don't) on top.
So yeah, I agree with you about CAP Theorem and stuff, disagree with Cockroach's particular choice - but they do have some pretty great detailed explainers/documentation on their view, and therefore should be treated seriously and not written off.
CRDB is also master-master, all the nodes are the same and can serve reads and writes. That has no bearing on AP or CP.
I know that this is meant to mean: in the real world you cannot just write off partition resilience and still call your system highly available, since partitions will happen sooner or later and when they do your CA system won't be available.
But in the other hand, having a system that is always available _except_ during a network partition is a useful thing: you can design a network where partitions happen much rarely that the rate at which individual machines die.
I.e. in practice a single node, while if you nitpick is the only true CA, will available for less time in average than a multi node CA system which if you nitpick is not CA (provided that the underlying network is partition resilient; it's not a boolean, it's a probability)
(See Google spanner)
I believe at the end of the day the CAP theorem is too fuzzy for such discussions.
(Fwiw their slo for multi-regional instances is 99.999%, although I have no idea what their measured performance is against that objective.)
"Does this mean that Spanner is a CA system as defined by CAP? The short answer is “no” technically, but “yes” in effect and its users can and do assume CA.
The purist answer is “no” because partitions can happen and in fact have happened at Google, and during (some) partitions, Spanner chooses C and forfeits A. It is technically a CP system."
which I believe is another way to word what I say in my post above.
It seems like a good solution for data that doesn't change too quickly?
there's a great talk on how its done with a custom gossip protocol implementation: https://www.youtube.com/watch?v=HfO_6bKsy_g
Cockroach DB: trade some performance for geographic redundancy. The trade off may work in your favor - e.g. read heavy workloads (or not).
I plugged in CDB I place of Postgres for some testing this week, was surprised it worked so well.
This is almost never the case, and CDNs are no exception. A CDN like Cloudflare that reuses your domain(s) means that they become just a useless point that your dynamic requests have to travel to and from the main server. A CDN that uses its own domains requires extra DNS queries, extra TCP & SSL connection setup, etc, plus it only starts loading when the browser has started processing the HTML.
There are also http headers that can instruct the browser to fetch cdn resource before the html is delivered.
This was a huge win for QoS on cache misses, which were a significant portion of our traffic. There are tons of tradeoffs to make this happen which is why we couldn't find an off-the-shelf solution to deliver the same results.
When partitions heal you simply merge all versions through conflict-free replicated data types. No ugly decisions, no sacrificing neither latency nor consistency. We call it strong eventual consistency [1] nowadays. And it's exactly like CDNs, except more reliable.
I'm wondering, since CockroachDB keeps lying about and attacking eventual consistency in these PR posts, the whole "consistency is worth sacrificing latency" mantra might not work in practice after all. People just don't buy it, they want low latency, they want something like CDNs, something fast and reliable, something that just works. Something that CockroachDB can never deliver.
[1] https://en.wikipedia.org/wiki/Eventual_consistency#Strong_ev...
What if two users want to change e.g. the telephone number of an existing record during a network partition.
There just is no obvious way to merge a telephone number. One of them is correct, the other is incorrect.
Can CRDTs solve my simple problem?
i.e. this is solved with CQRS and Event Sourcing and it probably works in like all databases. It's quite complex but pretty reliable, I'm pretty sure that everybody already built at least a extremly simple append only event log.
where you store (i.e. which database system, cluster whatever) your stuff doesn't matter.
Basically though that makes a CP tradeoff, since a network partition that hides the primary kafka replica from some of the writers causes writes to fail.
There’s simply no way to have a globally distributed system that’s always available and always consistent, either you get inconsistency or you drop writes.
all CRDT will give you a deterministic outcome and provide enough information to allow a user to decide if this outcome is acceptable
In your example, that might mean changing the single "phone number" field to many "phone numbers", so that merging the two writes results in a customer record with two phone numbers. This preserves the data, but pushes conflict resolution (which number should be used?) out into the consumers.
There are other DB systems with such properties "they" can use, e.g. Cassandra. But CockroachDB also gives ACID transactions, which may be important for others.
If the write has to be consistent and available across multiple regions, it will need to synchronously replicate that write to all the regions, thus incurring the same performance penalty as RDS or any other consistent database.
[1]: https://www.cockroachlabs.com/blog
[2]: https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
When your cluster spans multiple nodes (physical machines, virtual machines, or containers), newly split ranges (or more specifically, replicas of these ranges) are automatically rebalanced to nodes with more capacity. Writes addressed to a range are handled by the Raft leader for that range (which can hop around its various replicas as needed). Writes to different ranges (non-overlapping key spaces by definition) are processed independently, and very well may be processed across multiple machines.
Source: https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
WRT to the writes here, if a majority of the replicas for that range are in the proximate regions, the requests would only travel that far before responding. I believe the argument is that this a more flexible design point than a single point of entry for all incoming writes, regardless of the origin. The cost to write out to the furthest region within any majority of replicas is of course inevitable to have cross-region durability, alternatively you could trade this off to have the majority of your replicas specific to requests from a specific region, be located to that specific region.
So we are in agreement that CDB has same write latency as rds for multi region deployments. The article seems to imply that is not the case, but as you yourself agree it actually is.
To reiterate: I believe the argument is that this a more flexible design point than a single point of entry for all incoming writes, regardless of the origin.
While some of the technical underpinnings could be improved, the state of the internet is determined by social and legal considerations, not technical ones.
S3?
It would also be useful to me, in some cases, to be able to perform a read-only query in an "inconsistent" mode to avoid that cross-region latency, at the expense of potentially receiving stale data.
Might be helpful if you already know the couchdb answer for each feature.
Or more like federated SPARQL on RDF? https://www.w3.org/TR/sparql11-federated-query/
Never mention lower performance (even on single-node). Add to that aws-vps with pseudo-cores and spectre-upgrade and good luck with your tps-reports.
And I agree that this is perhaps one of the many situations where people throw away the "C" of "ACID" for no reason beyond it is modern to do so. (At least most strive for "Eventual Consistency", but that's another can of worms.)
This is even less understandable once you notice that PostgreSQL offers a lot more features than most other databases (SQL or NoSQL) and is extremely flexible and extensible - even if you use it just as a fancy JSON or XML store.