A Primer on Database Replication (2017)
brianstorti.com
brianstorti.com
If you're more interested in the quorum-ing tech that's come about (algorithms like paxos and raft) and the underlying distributed systems research I've recently tried to compile all the paxos family of algorithms in a blog post[0]. There has been a lot of work recently (notably EPaxos, WPaxos, SDPaxos) on tuning and improving the algorithms, including consideration for long distance WAN connections (that's what the W in WPaxos stands for).
[0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all
This is pretty good non-FUD post about distributed systems (unlike many posts related to consistency and consensus coming from distributed database startups and megacorps).
Although missing some important real world considerations in reasoning, namely latency in WAN setups (reducing number of RTTs isn't really about that).
> In the wide area, latency is dominated by network communication,which is decided by the number of round trips, and the distance tothe replica to contact. The test for Multi-Paxos is omitted becauseits disadvantage is obvious: client has to communicate with theremote leader, as long as it is not co-located with the leader.The replicas and clients for wide-area experiments are deployedin California (CA), Oregon (OR), Ohio (OH), Ireland (IRE) and Seoul(SEL). The sequencer of SDPaxos locates in CA. The round-triptimes (ping latencies) between these regions are shown in Table 1.
I think this case is even worse than most real world cases people are dealing with today. Google/Facebook might have data centers this spread out, but I feel like 90% of the people who read that post will be worried about multiple regions in America.
The takeaways, I hope people will get from this article is that there is no one solution which will work best in all situations. It is a design decision about what advantages your application needs to have and what sacrifices you are able to make. The more limited and strict you can be about how your application reads and writes data, the more options you have to make your database backend more robust. Applications that "try to have it all" usually just end up doing everything poorly.
The other takeaway is the the physical limits of the universe are the greatest barrier database systems are working against. You simply cannot get around them. When you end up working with a team who is unwilling to acknowledge this, run as fast as you can.
[1] https://www.allthingsdistributed.com/2012/01/amazon-dynamodb...
Of course, there are still tradeoffs, but for many applications, a distributed database with strong consistency guarantees is likely to be a good choice for many projects.
After having worked on these types of problems for a decade, and successfully running an Open Source database startup for nearly a half decade, they are certainly and only the right way to go.
They seem to suggest they aren't well adopted, and suggest there are severe limitations that are withholding their adoption.
This was true with append-only or log-based CRDTs.
But now, state-based graph CRDTs (like we've implemented in https://github.com/amark/gun ) have solved that. The only thing you can't do with them is Global Strong Consistency (think banking), but their Strong Eventual Consistency guarantees make them the best solution for literally everything else.
They are also run in production at large large sites, like the Internet Archive (top 300 site globally), D.Tube (1M monthly uniques), notabug.io (P2P reddit), etc. with GUN.
The article kind of makes this joke:
> (otherwise our lives would be too easy, right?)
But here is the kicker, both Internet Archive and NAB integrated/built in 1 week. Literally yes it makes people's lives easier.
And you should too! Next time you build a non-banking app, you should consider using state-based graph CRDT to cover 99.99% of your use cases!
- What is "easy" for you might not be for others.
- Banking systems don't use strong consistency. They can just rollback/swallow the costs of invalid transactions after the fact. Online mom-and-pop shop... maybe they can, maybe its better for the site to just have some downtime sporadically and avoid such hassles entirely.
- P2P is a nightmare for anything that requires audit trails (and there are legal requirements about that in many industries).
- There exists a whole ecosystem of databases dedicated to analytical processing workloads (OLAP). Graph databases are typically not the best at it.
- Conflicts happen. Your "conflict-free" datatypes embed the conflict resolution rules in the data structure. This is fine for some conflict resolution rulesets, but it cannot be done for some other sets (e.g.: in these weird circumstances, the user decides what the resolution is by clicking a button).