Hacker News Reader

Comment by bawolff | Hacker News Reader

I guess maybe i should read the linked papers, but i kind of dont get it. How do inherently conflicting updates get resolved? What happens when one writer sers bankBalance = 200 and the other sets bankBalance = 500; ? Or does just significantly limit the type of applications you can have?

View on HN

https://github.com/rqlite/rqlite might have the answer, it uses the Raft consensus protocol to solve time based conflicting issues like that.

rqlite author here, happy to answer any questions.

How hard would it be to release APE versions of the amd64 binaries of rqlite?

https://justine.lol/ape.html

No idea.

If a patch was provided, would you consider distributing the binaries on the github page?

(I can have a look, I'm just not very familiar with go intricacies)

Sure, would be happy to consider.

That isn't how CRDT's work though. In the example of mutating a bank balance, there would be addition operations and subtraction operations. So /eventually/ all nodes would arrive at the same balance. It's a bit like event sourcing but lower-level.

Depends on the type of CRDT. You can have a CRDT which supports setting a register (or column value in a row) to a particular value. This is what Crsql appears to do as one sibling comment indicated. There are a number of ways to resolve conflicting updates, but Crsql chooses LWW (last writer wins).

Right, but its how SQL works. So did this project get rid of SQL? What does it mean in context to join SQL and CRDTs together?

You don't update the account.ballance, you insert a new transaction for that account with the amount required. At eod, you calculate and update the correct balance.

That isn't necessarily "how SQL works". Sure, you might design your schema like that. But many accounting systems don't necessarily store a "balance" except perhaps as a performance optimisation. They derive the balance from the history of transactions that have occurred on an account. And a transaction is, essentially, a + or - value.

Indeed it can be argued that CRDTs are a natural fit for SQL because it is essentially a derivative of set theory

If you are essentially going to just have inserts that are not dependent on any reads, why are you bothering with fancy algorithms like CDRT? That is the trivial case.

I think the question is whether this particular CRDT has any notion of transactions to make sure the database is left in a consistent state after a merge.

As I understand it, the only guarantee we get from all CRDTs is that all copies end up in the same state irrespective of merge order. And then there are some trivial examples, like counters, where this state is also logically consistent. But that is not guaranteed in general.

So what about the classical example of transferring an amount from account A to account B if account A has sufficient funds?

I think for a CRDT to support such cases (for arbitrary schemas) it would have to have some notion of atomicity and some way to specify preconditions.

I don't believe it is possible to arrive at any generic solution that resolves the kind of conflict that you are describing. The best they can do is keep the latest value at row+column level.

Notwithstanding, there are a lot of possible changes that can compose well, but with traditional databases it is hard to merge divergent datasets even if the changes are non-overlapping. That is where solutions like this improve the state of the art - by obviating a need for secondary sync/merge layer on top.

For the actual cell level conflicts the application developer will have to either write custom resolution or model the domain so that they can be resolved by user later.

Sure, but i would kind of expect the readme document to explain what happens (what is the consistency model) since this is not an obscure case but basically the primary use case for an ACID database like sqlite.

https://github.com/vlcn-io/cr-sqlite/tree/ca9a67f9050844458e...

It's not very clearly stated, but the links show it has "Last Write Wins" semantics for conflicting changes

https://bartoszsypytkowski.com/crdt-map/#crdtmapwithlastwrit... https://bartoszsypytkowski.com/operation-based-crdts-registe...

It's complicated, but you can grok it: https://en.m.wikipedia.org/wiki/Conflict-free_replicated_dat...

The part i dont understand is as far as i understand, CRDT requires data types with commutative update functions, but SQL does not have that. The part i dont understand is how these two technologies that seem to have conflicting requirements are being joined.

Columns are LWW (last writer wins) registers within rows which are LWW maps. When concurrent writes occur because peers may not be communicating, there is typically an arbitrary but convergent selection based on peer id.

There's different strategies for solving conflicts but CRDTs wouldn't be a good fit where you need a total ordering of all transactions -- such as bank deposits.

You can handle bank deposits with a counter CRDT.

https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

Reply on news.ycombinator.com