Not to detract from the article, which I found interesting for a number of reasons - both in terms of the technology and future of that technology as well as the personal realization of "the thing I'm building won't be necessary one day".
More generally, CRDTs are useful when syncing any kind of change-events between peers (with no central ability to pre-linearize those events), where the ordering of change-events doesn't matter — i.e. where the operations described by the events are "commutative", such that you'll get the same results out if you apply events {1,2,3} or {3,2,1} or any other order. As long as all peers see all events, all peers arrive at the same eventually-consistent position in the "lattice" of possible states. In other words, CRDTs + gossip protocols = the ability for peers to just process events as they hear about them, without a worry about receiving events from peers "out of order"; rather than needing to wait around for a some elected peer to forcibly linearize the events (i.e. without the need to introduce a serial write bottleneck.)
A good example of the distributed-systems-backend kind of use of CRDTs, is in the Elixir web framework Phoenix, which does a sort of mesh delivery of web-socket messages between a cluster of backends. CRDTs are used to sync presence information (= which clients are connected to which backend) between the backends.
edit: Oh you edited your post there's way more now, but I can't read it all now x_x but fwiw when I said "anyone else" I meant that I use crdts.
I never used it (darcs) but got the impression that it was easy to get into merge cases that were very hard to resolve.
(To be clear: I am not /claiming/ the above, I am asking whether my recollection is correct)
It's going to take some time for this to all come together, but it's exciting.
[0]: https://news.ycombinator.com/item?id=29978659
There's more great CRDT writing which deserves some attention. For example, this series by Bartosz Sypytkowski is long but excellent:
https://bartoszsypytkowski.com/the-state-of-a-state-based-cr...
He talks about CRDTs from the ground up, from much more of a mathematical / FP point of view. I learned a lot reading this series!
Why have I been focussing on collaborative text editing? Because I think performance has been the #1 blocker stopping us from using CRDTs in most software. And text CRDTs are a canary in the coal mine for performance. Uh, thats a tortured metaphor. I mean, If we can make text CRDTs fast, we can make everything fast.
And JSON editing (what everyone actually wants) needs to embed text editing anyway. So we've sort of gotta do that first anyway.
Smart people have also been working on the JSON editing problem. I'm a big fan of Greg Little's Shelf algorithm[1] for last-writer-wins JSON editing (which is what people usually want). Shelf is simple enough you can implement it in less than 100 lines of code. But shelf doesn't support collaborative list or text editing (text is a list of characters). We need all of these pieces, then we need to mush them all together in a sensible way. And thats something Yjs and automerge both already do, and do quite well.
I’m more excited about the use of CRDTs outside of text editing, there are so many use cases. However there are still some problems to solve with the existing general purpose toolkits such as Yjs and Automerge. Both give you simple data structures and some more complex structures specifically for rich text. For a true conflict free system you need the CRDTs to match your data model exactly so you can never have a invalid state after a merge. This is almost never the case with existing toolkits.
Take Yjs and it’s ProseMirror bindings, Yjs can represent any internal state of a ProseMirror document and merge them. However it does not have an understanding of the specific ProseMirror schema being used, this could limit the valid states possible (a max length on a heading for example). When Yjs merges multiple edits you could end up with an invalid document for your schema, Yjs loads this into ProseMirror which then throws away the invalid data (with the heading example it just deletes the excess text, it may be better to dump it into the next line but should be configurable).
These sorts of problems are “ok” in a text editor but would be much harder to cope with in a database or a 3d cad tool.
Ultimately the general purpose CRDT toolkits need to have more internal types with a wider ability to capture intent. I also think they need to have the ability to understand the schema of the data structure they are tracking and how to handle more complex merges.
The alternative to doing that with a general purpose toolkit is to build your own CRDTs from scratch for your data model. I don’t think many people would go that route though.
Once those sorts of problems have been solved I think we will see their use explode into new areas.
It’s still early days with them but exciting time ahead.
https://www.figma.com/blog/how-figmas-multiplayer-technology...
Most of the applications I worked on in the last decade (web saas crud stuff) are dominantly transactional with maybe a thin slice of collaborative functionality. I suppose the next decade could look different but I don't see why that would be the case.
The thing i've found most interesting is it appears[2] that CRDT backends need to expose CRDT flavored types to users. Which is to say how i'm writing this combines the notion of a type, say `[i32]` with how you want the merges to work. CRDT works great but based on my amateur-hour researching on the subject i don't feel you can write a single CRDT merge strategy for a single data type ala `[i32]` and have it be always correct. Applications need to indicate enough context on what makes sense for a given data type.
So yea, i agree with you. I'm interested in making a database-like thing, backed by CRDTs, but i also have seen very few general purpose implementations with CRDTs. It feels like i'm breaking "new ground", while having no idea what i'm doing and having no intention of being an actual researcher here. I'm just making apps i enjoy heh.
[1]: https://github.com/attic-labs/noms [2]: appears, because i could easily be wrong
Ideally when clients (corporate overlords) would send product data an event streaming broker (like Kafka) would log the product. One consumer would insert the product data into the SQL database (current implementation), one would generate CRDTs based on diffs from the current product model, and a chained consumer would broadcast the CRDTs to all connected clients.
The initial sync would be brutal so we'd still support server side queries and rendering but eventually the customers could generate the PDFs themselves regardless of if our server was busy (or offline which happened often).
It was a fun project and I helped lay a ton of the ground work but due to company politics and a generally toxic workplace I jumped ship. My new job would never need CRDTs so I'm now just sitting back and watching the space evolve.
I'm finding it painfully difficult but it is evolving steadily.
- Google Docs
- Microsoft Office 365
- Figma
- Quip
- Notion (my employer)
They have had review features that are decent. But... Best feedback I get is still typically verbal or otherwise out of band.
In the beginning was the word.
A lot of things can be expressed using text.