CRDTs are the future
josephg.com
josephg.com
We went with OT for our real-time syncing of edits in 2010 and a decade later, we are still sticking with OT for reasons I already stated sometime back - https://news.ycombinator.com/item?id=24186883
However, in the spirit of "There are no solutions, only trade-offs" CRDTs are absolutely necessary for certain type of syncing - like syncing a set of database nodes.
But for systems which already mandate a central server (SaaS/Cloud) and especially for a complex problem like rich-text editing (i.e semantic trees) I still think OT provides better trade-offs than CRDT.
I respect Joseph's conviction on CRDTs being the future, so I guess we'll figure this out sometime soon.
In the link, OT is aliased to "Operational Transformations"
Source: OT's Wikipedia article
But I felt the same. Never heard of "Operation Transformation" before and both OT and its alias were equally opaque to me.
A central server: Most OT algorithms depend on a central system for intention preservation. CRDTs are truly distributed and need no central server at all.
Memory: Traditionally CRDTs consume more memory because deletions are preserved. OT lets you garbage collect some operations since a central system is already recording those ops and sequencing them as well.
Analysing and cancelling ops: OT lets you easily analyse incoming ops and modify/dummy-ify/cancel them without breaking the consistency. This convenience is not necessary for most cases, but really important for rich-text editing. For example when someone merges a couple of table cells when another user is deleting a column, we need to analyze these operations and modify them so as not to end-up with an invalid table structure.
If you’re going to invest your complexity budget somewhere, it seems like this is a good place for companies dealing with these structures.
A real world implication is that if you want to add a new operation to a system (like, table column merge, or moving a range of text), with OT, you can probably find a way to extend what you have to get it in there, with a painfully non-linear cost as you add more new operations. With CRDTs, you may find yourself entirely back at the drawing board. But the stuff you do support, you will support pretty well and reliably...
Personally, I prefer CRDTs for their elegance, but it can be difficult in a world of evolving requirements
Exactly the experience that we have in CKEditor 5. OT is awfully complex. However, we can fine tune it for new operations and different scenarios of usage (e.g. undoing changes is one of them). With CRDT, at least to my understanding, we might've been limited to what we'd took into consideration on day one. Seeing how many things we learnt very late in the process (5 years so far, some documented in https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri...) makes me thing that we'd sink the company twice already.
I worked on a prototype for a couple of months which would have been a partial competitor to CK. We went with CRDTs because it was quicker to get something simple and reliable off the ground to demonstrate, but we were pretty sure we'd have to switch to support something for the longer term.
It wasn't chosen to go forward as a focus of the business, but it was one of the most fun periods of commercial programming I've done :)
The downside is that if a single document gets a spike of traffic because it goes viral, then that document won't scale.
yjs[2] is one CRDT that handles text reasonably well, but it can still run into performance edge cases (as they plainly/honestly admit in their README).
[1]: https://github.com/automerge/automerge/issues/89 [2]: https://github.com/yjs/yjs
One thing I would love to see is the addition of wildcard addresses like the way google has and microsoft added (user+site_string@domain.com).
Thanks for your hard work on a great product!
Frustratingly though, there are so many features heaped in that there is no cohesion. Things are frequently buggy, unreliable and disjointed. I’d almost be able to forgive it but unfortunately the support is really terrible too.
I assessed a lot of crm software and each one I kept finding things they didn’t have that zoho had but for the reasons above we ultimately chose something else. Which is a shame, because I would pay them a lot more than they ask, for them to just be a little better.
This is my experience. After years of paying for the service, for multiple users, using tools that seemed to have been forgotten by the dev team, my credit card failed to process one month.
Support was not interested in helping me. Not at all. Ended up having to move my team's workflow off Zoho. I had been begging them to bill my card, but they just kept sending me generic emails asking me to try again. Phone support was even less helpful.
My colleague wrote down our thoughts in https://ckeditor.com/blog/Lessons-learned-from-creating-a-ri....
The other aspect of CRDT that's in some scary is what @mdpye explains in https://news.ycombinator.com/item?id=24621113. With OT, if we get the corner cases right, we can tune and extend the implementation. With CRDT that might be game over.
I'd like to have things like CRDTs under my radar to pull from when architecting technical solutions. But I don't.
I have been coddled by the bullshit of "just-good-enough" web development for nearly a decade, and I feel like I will be haunted by it forever at this point.
I WANT to employ mathematically proven solutions, but not only has it been 10+ years since studying computer science (and with it, the relevant mathematics) but I never even graduated.
So here I am, another under-educated over-paid GED holding Rails engineer putting together shit solutions I know will eventually leak.
So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start? I feel like going back to school full-time is out of the question, because I have a life now. Maybe online classes? I'm curious to hear thoughts from people who maybe came from a similar pit of despair, or otherwise understand what I am talking about.
It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again. Is this just the profession I have locked myself into?
Turns your front end app multiplayer with a library.
They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.
Perhaps it would be prudent to read the actual article before tossing off this sort of comment since it's indeed the whole point?
One of the classic problems was size and performance. However both problems have now disappeared. Operations are `O(log(n))` and the size overhead is a factor of 1.5-2.0. That's quite workable.
I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating.
I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.
I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work.
I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.
Get your intuition down really good and 10+ years is not required.
The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future.
The material scientist may hit a dead end, or their company may axe their entire department to "cut costs".
One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded.
I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.
Meanwhile, I feel like I'm an adult and can make my own decisions. I don't care if what I like is not "cool" or "hip" or some "up and coming technology". I don't have to be in research or the cutting edge of something to make interesting personal discoveries.
In my career I've accidentally stumbled upon multidimensional OLAP cubes (in the form of SQL Server Analysis Services) and discovered that they're quite fun to work with, actually. The reputation of business analytics as something "dry and boring" is undeserved. It's a wonderful mix of human problems like "manufacturing calendars" and quirky sales bonus calculations on top of fancy optimisation tricks to make calculations over terabytes of data nearly instant.
I've also had the incredibly gratifying experience of making a bog-standard web app with only a few mildly tricky technical bits for a department of education. It helped about a million kids read more than they would have otherwise. That's something. In fact, I'm certain that my one library app made kids read more books they wouldn't have otherwise than a dozen English teachers.
My parents were educators, and I outperformed them a hundred fold without ever stepping foot in a classroom as a teacher.
So yeah there's no shame in working in more mundane & utilitarian areas. And, like you said, you might reach more people with your work that way.
Wave was disbanded, but the technology was apparently integrated into other things such as Google Docs, which uses OT for collaboration quite succesfully (despite the criticisms in OP's article).
Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all.
Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way.
Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.
See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move.
Good luck. I believe in you.
To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.
Now that I have accepted that my lasting happiness will not be found in a prestigious job or fame as an engineer, where shall I find it? Perhaps the same place other humans have for the last ten thousand years. In the center of a quiet meadow in the woods. Next to an animal that considers me a friend. In the arms of someone I love. Plucking strings on an instrument, singing my stories.
Is this response a coping mechanism, or an acting choice to minimally engage with the parts of human existence which entail naked power brokering and empire building? Maybe it's a little of both. May we both find our own peace somehow, friend.
HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms.
Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly).
For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.
> So, having said all of that, is there hope for adding things like this to my toolbelt?
Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know.
We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.
Based on everything I've seen about them, OT/CRDTs sound like they are that.
I recommend implementing them for the fun of it. I love it. But don't put anything you write into production until you can leave a randomized test suite running overnight without errors.
Example:
Random op generator: https://github.com/ottypes/text-unicode/blob/master/test/gen...
Which powers this fuzzer: https://github.com/ottypes/fuzzer/blob/master/lib/index.js
You just replied don't. Have you considered he wants to this regardless of not applying them purely commercially and wants to perform research, or in a novel industrial capacity?
> It's just... sad to know enough that I produce less than optimal work but don't know enough to confidently prevent it from happening over and over and over again.
Basically, the notion that not knowing how to implement this means the resulting product is sub-optimal. My counter-argument is that delegating work that needs expert domain knowledge and skills to the experts and use the fruits of their labor will result in a better product.
[1] https://blog.cloudflare.com/introducing-workers-durable-obje...
I'm sharing the vision with you that things should just work. It literally takes you two lines of code to make any of the supported editors collaborative. The Yjs types are a minimal abstraction over the crdt model. I hope that other people build more abstractions around Yjs (for example a react store that simply syncs).
The thing is.. all of what you are talking about is already possible. A wasm port will eventually bring some performance improvements. But Yjs is already very close to optimal performance and it works in all browsers.
But I really am interested in more feature request. I'm mainly interested in building collaborative apps with CRDTs. CRDTs as a data model is an entirely new concept and we need to get experience on how to build apps with CRDTs. And also we need to find out what is still missing.
So head over to discuss.yjs.dev and share your experience.
- Support for moving objects (as well as inserting and deleting)
- Better support for long lived documents. (I want to be able to pull historical deleted stuff out of a document so it doesn't grow unbounded).
- Presence information and markers (this is sort of supported now?)
- The gorgeous automerge API frontend ported across
- Support for interacting with yjs from rust / C / swift / etc
I want all of that too. Although I would add Automerge's API on-top of Yjs as an additional layer. It really does play nicely with Web Frameworks like react. Although for building editor bindings I would choose the Yjs types because they allow faster access to the data. Yjs only provides a minimal abstraction around the CRDT. I hope that other people will build nice stores/abstractions around Yjs that make interacting with shared data easier.
- Presence information and markers (this is sort of supported now?)
Presence is handled by a different CRDT (the Awareness object in y-protocols). All providers implement the Awareness protocol. In many cases you want one awareness instance for several documents, so it makes sense to separate them.
Markers is probably what I refer to as relative positions: references to specific positions in the document. It even plays nicely with undo-redo.
- Better support for long lived documents. (I want to be able to pull historical deleted stuff out of a document so it doesn't grow unbounded).
Yjs does this reasonably well. To a point that you can even build 3d applications like relm.us with Yjs. But you are right that more work could be invested in researching even better solutions.
- Support for moving objects (as well as inserting and deleting)
Moving of elements can already be modeled with Yjs [1]. Although the approaches described are not very convenient to use. Automerge implements this feature natively, but I'm a bit hesitant to implement the next best approach because I see many downsides that conflicts with features I plan to implement. There is a discussion thread where I outline a few of the tradeoffs [1].
I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0].
And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore.
If you need to understand certain things at a deeper level, then explore the classes and other guided tuition.
I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft.
You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.
I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas.
I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough.
That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!
In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory.
I included all the resources I used to learn the concept on the page too.
It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources ).
Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.
Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)
There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.
[1] is an excellent tutorial that assumes no initial familiarity with CRDTs or the math that underpins them. It walks you through both the formalisms and the implementation, which is pretty key to understanding why making real-world CRDTs flexible enough to handle things like rich text editing is hard.
[2] is a talk that goes more in-depth on the hard parts
[3] goes deeper on OT vs. CRDT
It's worth noting that many of the CRDT discussions focus on collaborative text editing. That's a really hard problem. CRDTs are (and have been for some time) a useful primitive for building distributed systems.
[1] https://news.ycombinator.com/item?id=23737639
Or, if you'd rather not, can you share some examples of problem domains, that we at home can search for?
There's also 2-year old list of CRDTs used in prod [2].
I know Azure CosmosDB (NoSQL) advertises it's own use of CRDTs for global replication.
[1] https://www.infoworld.com/article/3305321/when-to-use-a-crdt...
What I find most interesting about it is that it has reduced the state of multiple 'smart' user-facing widgets/apps into a common, lowest-common-denominator format (a text document) that lends itself more easily and intuitively to collaborative editing and CRDT operations.
I don't know for sure whether this is the path forward for CRDT-based applications in general, but I think there are valuable ideas there. It does raise the possibility of the widgets/applications occasionally being in 'invalid' states; but rarely in a way that the human participants wouldn't notice or be able to fix themselves.
Whether that scales to the complexity of the state management for a multi-track audio editing session, I don't know; but it could be instructional to compare.
The approach we use at Jackbox for making the state of an xbox game mutable to thousands of live viewers on twitch is to have lots of little CRDT values, mostly just counters and sets of strings, and you merge the little values independently of one another, which is very different from the situation of editing a text document, which is typically structured as one big value. I wonder if, for a DAW, you could merge at the track or control level instead of the workspace level. E.g., communicate as an independent value the state of an individual fader, and communicate either states or operations on that fader and have each client merge them. In this example, the fader's state would be encoded as a PN-counter with a range condition, and you'd replicate increment and decrement options, like it was a networked rotary encoder. So every mutable thing in the DAW would be a value having operations that can be merged, instead of having a single big value representing the entire state of the DAW. My use-case is also funky because I have potentially thousands of writers writing to the same key, but only a single reader, and the reader doesn't need an update at every change, so I use state-based CRDTs, but I think most other people using CRDTs use operation-based CRDTs. Also not sure how you would mutate two separate values transactionally or if that's a thing you even need.
There are lots of mutable things in a DAW that are not numeric parameters.
The state of a playlist (just think "some time ordered list of objects") is not treatable in the same way as the value of a fader.
If you had a context-aware XML parser and access to timestamps for every XML node, you could do the human-aided merge by considering each node and just using the latest version of the same node, falling back to the human when there's a deletion conflict (for example). But this doesn't actually merge the attributes of a node or deal with conflicts between otherwise sequential edits.
I see text editing as the beachhead for this tech. Text editing is hard enough that other systems don’t work well, so you’re kind of forced to use OT or CRDTs to make it work. And text documents are lists of characters - so once you’ve made it work there you have an implementation that can also sync arbitrary lists. At that point, editing maps (js objects) and adding support for arbitrary moves and reparenting will allow us to add real-time editing to a lot more software.
I think there’s a lot of interesting software architectures that can be enabled by making everything in a system a CRDT document. But text is still the beachhead. And a good litmus test for performance. And an important application in its own right.
However in my experience most games's shared world state cannot be reconciled if one player's action is inserted in time prior to another player action. Or future actions reveal hidden information.
Even turn based games have turn timers. Including Jackbox games!
Among public databases there may only be two correct CRDT implementations in the world - Cassandra's and etcd's. Compare to how many database products make CRDT or CRDT-like promises, and how few probably actually work.
Lots of software promises CRDTs-like-over-actors, e.g. Erlang, the new Cloudflare product. Not really qualified to talk about them.
Most applications normal people use boil down to chat conversations - chat apps, Gmail, Facebook, collaborative text like you said, all sorts of log-like things - or serial, recommendation-driven consumption - YouTube, Netflix, which use incremental / batched cells in a matrix to update their recs. This stuff is insensitive to order and time. Then there's e-commerce, which is inherently linear, centralized, single-actor, all-or-nothing transactions, CRDTs provide no advantages here.
It's tricky. On the one hand you'd get this interesting discussion of CRDT applications. On the other it may wither under intense scrutiny.
So yes, I agree that CDRTs are indeed a promising endeavor.
We released the our collaborative Jupyter notebook in 2014 as a plugin to Jupyter classic. We then iterated on what we learned over the years, completely rewriting everything multiple times, including the entire realtime collaboration stack. Cocalc's Jupyter support is pretty mature and battle tested at this point, and also includes a TimeTravel slider that lets you view all past versions of a Jupyter notebook and integrated chat.
I was a college professor (at Univ of Washington), I started a company around this in 2015, so CoCalc has soo far been mainly aimed at serving the needs of academics teaching courses. It's been increasingly popular lately, e.g., in the last month over a half million distinct Jupyter notebooks were edited on https://cocalc.com. Of course, many of these notebooks are homework problems. Anyway, our company is doing very well, and we hope it will eventually be a "10M startup opportunity". :-)
In particular I've been wondering about the space between CRDTs and the 'theory of patches' such as we discussed with Pijul the other day.
I have a collaborative editing project that's been sitting in my in-box for a long time now because I don't want to write my own edit history code and existing tools don't have enough ability to reason about the contents as structured data. The target audience is technology-averse, so no 'dancing bears' are going to interest them. It's not enough for it to work, it has to work very well.
Version control works wonders with line-oriented stuff, which covers more or less every programming language in existence.
It doesn't do so well with non-line-oriented structured formats such as XML (not sure how JSON or TOML) fits in here).
Given that collaborative editing typically works with non-line-oriented data formats, you can see the issue, I think.
Even git allows for pluggable diffing, and doesn't force line orientation. What's missing is the concept of moving something, as distinct from deleting lines/chunks and then inserting lines/chunks which just happen to be the same.
This is not a problem which CRDTs have, to put it mildly. I believe pijul understands it as well. A lot of this stuff is right out on the cutting edge, and as it matures it will become practical to connect the edges, such as a CRDT which collaborates with a parser to produce grammar-aware patches which are automagically fed to pijul or something like it.
This comes with a host of problems, mostly that we're not used to dealing with a history which has this level of granularity, most of which we don't want to see, most of the time. But they would be nice problems to have.
That this is generally a feature of the diff tool and not the version control is a bit disappointing.
There's a next level of VCS forming on the horizon, in some combination of CRDTs, patch theory, and grammar-aware diffing.
Which should also learn from fossil, and consider metadata such as issues and surrounding discussions to be a part of the repo.
A really robust solution would also be aware of dependencies and build systems, and even deployment: I see these as all fundamentally related, and connected to versioning in a way that should be reflected and tracked through software.
The Linus' original insight that made git successful in part was exactly the fact that: You don't need to do the delta scheme on the VCS-object level, but you can just use it on the storage/compression level. So conceptually CRDTs are more similar to the VCSes of the past (e.g. SVN) than they are to git.
Much of real-time collaboration goes back to networking and real-time networking used in distributed multi-user systems like games, where simulations need to sync on a server. In games though, Dead Reckoning [2] is used as well as interpolation and extrapolation in prediction, much of it can be slightly different for instance with physics/effect, but messages that are important to all like scores or game start/end are reliably synced and determined on the server.
[1] https://visualstudio.microsoft.com/services/github-codespace...
[2] https://www.gamasutra.com/view/feature/131638/dead_reckoning...
The problem spaces are quite different. Problems that emerge on a minute-by-minute basis in collaborative editing emerge on a week-by-week basis in source control, and when the problems emerge in the latter, they tend to be much larger (because you can build up a much bigger merge conflict on a routine basis with the big chunks you're making).
Yes, it's true that if you squint hard, it looks like version control is a subset of collaborative editing, but I'd be really hesitant to, say, try to start a start-up based on that observation, because even if we take for the sake of argument that it's a good idea to use the same underlying data structures, the UI affordances you're going to need to navigate the problem space are going to be very different, and some of the obvious ways of trying to "fix" that would be awful, e.g., yes, you could give me a "collaborative space" where I see what everybody's doing in their code in real time... but it's a feature, not a bug, that when I'm working on a feature I'm isolated from what everyone else is doing at that exact moment. When I run the compiler and it errors out, it's really, really nice to have a good idea that it's my change that produced that result.
(I'm aware that collaborative editing also has the "I was offline for a week and here's a bunch of conflicts", but I'm thinking in terms of UI paradigms. That's not the common case for most/all collaborative editing systems.)
[1]: Not saying the only solution is the one we had now. A magic genie that watched over the code and made commits for you at exactly the right level of granularity would be great, so you'd never lose any useful context. But key-by-key isn't that level of granularity.
The bits where they don't interact also don't conflict. The bits where they do, look a lot more like collaborative editing.
They're also the spots where merges usually go wrong.
Beyond that it's insane. You do not want that in your version control system, as something built in, working all the time, across your entire team. It would be a massive anti-feature that would nuke your product.
Again, anyone thinking this sounds like a totally awesome idea, I strongly encourage you to try out the simple version, availablbe right now, of just "five or six people editing the same source code checkout" right now, before betting a start up on it. I guarantee a complete lack of desire to productize the result if you try it for a week or two.
The main difference would be that the PR check only compares with master, and you'd want to compare all branches. But a low-constant-factor test for collisions would make that cheap enough (conflicts can only occur for commits that contain the same file names, which is an O(mlogm) set intersection test of short text strings)
GitLab also detects if there is a conflict on a merge request as things change and will notify the user
Git does not work well for text because we have not figured out a nice format for text yet that developers and other people both enjoy. Developers want to stick to plain text as their format because we have so far failed to create nice tools and formats for structured data. Perhaps these affordances can appear thanks to a popularization of real-time collaborative editing.
Eclipse used to (?) have a feature where each local save was stored separately, so it was trivial to view local history.
I've done remote code reviews with shared documents.
I can certainly imagine a system that combines the two of these things into a single system where real time version control was integrated into a multi-user system.
I count five or six people, including the author, in your 'nobody'.
If you watch any of the videos on CRDT, live editing (real-time) is every bit as much a corner case as the 'play by mail' scenario that code repositories do.
Asynchronous communications complicate Last Writer Wins algorithms, because the network is partitioned and time stamps don't have the same predictive value. Git is DVCS, where the "D" really means partition-resilience.
So we ended up essentially implementing what you laid out, an in-memory revision control system, although using a bit more formal methods to reason about divergence/convergence of clients. The most basic operation was the "diamond merge": given operation x:A->B, y:A->C, construct x':C->D, y':B->D such that x' . y == y' . x It also had to satisfy certain other algebraic laws, notably diamond composition, which allowed us to compose these merging operations whenever we wanted, guaranteeing that the clients will eventually converge to the same data state. It was quite neat! Shame that it's all proprietary.
Good old days. I remember, the most pesky operation was implementing a good undo-redo algorithm, it's quite tricky, even once you add inverses.
https://github.com/ottypes/json1/
(Designed to be used with sharedb or similar.)
People think of OT / CRDT as realtime algorithms for realtime collaborative editing because they're always programmed and used that way. But the conflict resolution approach doesn't have to merge everything as-is. You could build a CRDT or OT system that generated VCS-style conflicts if concurrent edits happen on the same line of code. To make it a valid OT / CRDT algorithm the main constraint is just that every peer needs to resolve conflicts the same way. (So if I merge your changes or you merge my changes, we end up with identical document states). It would be easier to implement using OT because you only have to consider the interaction between two peers. But I think its definitely doable in a CRDT as well.
I think having something that seamlessly worked in both pair programming setups and with git style feature branches & merging would be fantastic.
I have a lot of thoughts about this and would be happy to talk more about it with folks in this space.
I also thought one could differ between two different kinds of conflicts. I call them simultaneities and disagreements. Simultaneities happen when concurrent changes happen and could be handled by CRDTS for example. Disagreements are conflicts between layers.
The idea is then that you can choose to work in the same layer as someone else, if you are working close. You can also "shield" yourself from changes in other layers by working in a different layer. If you want to detect a disagreement between your layer and a layer someone else is working on, you project your layer on top of that layer.
Even though I believe in these ideas I don't know how to get other people interested in them. It might be that they are flawed in an obvious way not apparent to me.
What would it take to make someone of your caliber curious?
> What would it take to make someone of your caliber curious?
Talk about it more! Write a blog post describing the use cases and how it would work. Make UI mockups. Talk about what problems exist today that your idea would solve. Hop on to the automerge slack channel (linked from the automerge github repo) and show us.
And if you want to, build it. You don't need anyone's permission to make useful software.
[1] https://roomservice.dev [2] https://news.ycombinator.com/item?id=24158509
Disclaimer: I've invested in roomservice.dev (and very excited about what they're building!). No affiliation with Tella.
CRDT as a service.
The core argument of this article: that CRDTs now work and distributed is better than centralized I question. I certainly want more distribution than "everything is run on a google server" but do I really foresee a need for distributing a single document? One server with an optimal OT implementation can probably handle near a million active connections.
In practice, that's plenty. Each piece of data having one owner is quite reasonable. There are lots of pieces of data.
I remain on the fence for collaborative text editing. Though it's great to see all the work pushing CRDTs forward!
Does it make sense for us as an opensource community to invest our time and energy making one really good CRDT (with implementations in a few languages). Or does it make sense for us to distribute that energy between a bunch of CRDT and OT implementations, with different performance tradeoffs?
My take is that its been hugely beneficial to us all that JSON is a standard, because I can use it from every language, and have confidence that the implementations are fast and good quality. I think we have an opportunity to make that for a good CRDT too. Even if OT would work fine in your architecture, if we have a great, capable, fast CRDT kicking around, it could be a reasonable default for most people. And my claim is that the performance difference between CRDTs and OT is smaller than the difference between high and low quality implementations. (I expect a well written CRDT in wasm will outperform my OT code in javascript.)
I also long for this future. I want all software to work in this manner.
Interesting to know how CRDTs are enabling this, or how the limitations of OTs have restricted development of these tools (although git and GitHub exist regardless)
edit: updated 'you are doomed' to 'you may be doomed'.
Believing there is a silver bullet is a fool errand.
From what I've read about CRDTs, it seems difficult to escape the overengineering trap when dealing with them.
I believe it's better to focus on kits of parts--API's and/or self-contained functions--that can be combined or ignored as needed, along with a variety of reference application samples.
Having lots of ways to easily filter and sort content is also very useful. For example, filtering and/or sorting annotations by person, group, date, content (sub-strings) is very useful. A query-by-example kind of interface is nice for this.
The ideas do look nice. And I suspect it has gotten farther than I give credit. However, sequencing the edits of independent actors is likely not something you will solve with a data structure.
Take the example of a doc getting overwhelmed. Let's say you can make it so that you don't have a server to coordinate. Is it realistic to think hundreds of people can edit a document in real time at the same time and come up with something coherent?
Best I can currently imagine is it works if they are editing hundreds of pages. But, that is back to the basic wiki structure working fine.
So, help me fix my imagination. Why is this the future?
So yes, hundreds of people can edit a string and produce a coherent result at the end. Contiguous runs of characters will stick together and interleave with concurrent edits.
I should also note that even if you use regular merge, and the end state of a text document is a complete mess after a refactor + concurrent edits, there’s enough data in the tree to simply pull out any concurrent contributions. They could then be reapplied manually if needed. Perhaps the app could even notice this automatically and provide an optional UI for this process. Similarly, it would be possible for the concurrent editors to remove the refactor edits and thus “fork” their document.
Comparing to git (as others have done) is interesting. The expectation is any merge is manually tested by the user. Such that it is not just the git actions at play, but all support activity. That is, the user flow assumes all intermediate states are touched and verified by a user. Where this is skipped, things increase the risk of being broken. (Is why git bisect often fails projects that don't build every commit.)
Same for games. Some machine gets to set the record straight as to what actually happened. Pretty much always. The faster the path to the authority for every edit, the higher chance of coherence.
With hundreds of authorities, machine or not, this feels intractable.
(But I think this would be unnecessary for most instances of real-time collaboration, since people tend to own and edit small portions of a document instead of jumping around and making drastic revisions all over the place. In fact, it should be very rare for two changes to actually be concurrent, unless an offline version of a document is being merged in. I would agree that "100 people editing the same document offline and merging periodically" is a less than ideal use of CRDTs, but I think they could offer many benefits even in this scenario, especially if paired with a central ledger as described above.)
The result may often be coherent at the sentence level if the edits are normal human edits, but often will not be at the whole-document level.
For a simplistic example, if one person changes a frequently-used term throughout the document, and another person uses the old term in a bunch of places when writing new content, the document will be semantically inconsistent, even though all users made semantically consistent changes and are now seeing the same eventually-consistent document.
For a contrived example of local inconsistency, consider the phrase "James had a bass on his wall." Alice rewrites this to "James had a bass on his wall, a trophy from his fishing trip last summer," and Brianna separately chooses "James, being musically inclined, had hung his favorite bass on his wall." The CRDT dutifully applies both edits, and resolves this as: "James, being musically inclined, had hung his favorite bass on his wall, a trophy from his fishing trip last summer."
In nearly any system, semantic data is not completely represented by any available data model. Any automatic conflict-resolution model, no matter how smart, can lead to semantically-nonsensical merges.
CRDTs are very very cool. Too often, though, people think that they can substitute for manual review and conflict resolution.
If two different git clients each implemented some automated form of merge-conflict resolution; and then each of them tried to resolve the same conflicting merge; then each client might resolve the conflict in a different, implementation-dependent way, resulting in differing commits. (This is already what happens even without automation—the "implementation" being depended upon is the set of manual case-by-case choices made by each human.)
CRDTs are data structures that explicitly specify, in the definition of what a conforming implementation would look like, how "merge conflicts" for the data should be resolved. (Really, they specify their way around the data ever coming into conflict — thus "conflict-free" — but it's easier to talk about them resolving conflicts.)
In the git analogy, you could think of a CRDT as a pair of "data-format aware" algorithms: a merge algorithm, and a pre-commit validation algorithm. The git client would, upon commit, run the pre-commit validation algorithm specific to the file's type, and only actually accept the commit if the modified file remained "mergeable." The client would then, upon merge, hand two of these files to a file-type-specific merge algorithm, which would be guaranteed to succeed assuming both inputs are "mergeable." Which they are, because we only let "mergeable" files into commits.
Such a framework, by itself, doesn't guarantee that anything good or useful will come out the other end of the process. Garbage In, Garbage Out. What it does guarantee, is that clients doing the same merge, will deterministically generate the same resulting commit. It's up to the designer of each CRDT data-structure to specify a useful merge algorithm for it; and it's up to the developer to define their data in terms of a CRDT data-structure that has the right semantics.
For a codebase, unit tests could be the pre-commit validation algorithm. Then, as authors continue to edit the piece, they both add unit tests, and merge the code. In the face of a merge, the tests could be the deciding factor between what emerges.
Of course, unless you have conflicts in the tests themselves.
And here's the thing: Can 100 people edit a document, even in theory, and have it make sense? I think the answer is "no," with or without technology.
I'm sure there are other uses for these data structures, but shared editing is always the example I read about.
Somehow I think the answer is no. There is a reason we still have to manually drop down to a diff editor to resolve certain kinds of conflicts after many decades.
The problem with using a CRDT is the CR part: there are generally merge conflicts in version control for a reason. If your data type isn’t “state of the repo with no conflicts” or “history of the repo and current state with no conflicts” but something like “history of the repo and current state including conflicts from unresolved merges” then maybe that would work but it feels pretty complicated to explain and not very different from regular git. Also note that you need history to correctly merge (if you do a 3-way merge of a history of a file of “add line foo; delete line foo” with a history of “add line foo; delete line foo; add line foo” and common ancestor “add line foo”, you should end with a history equal to the second one I described. But if you only look at the files you will probably end up deleting foo)
See also: darcs and pijul.
For documents, you might represent this as a merged document, with merge conflict markup inside the merged document - similar to using version control and getting a merge conflict.
Also similar to version control, the merge conflict can be a form of data where a valid CRDT operation is for a user to resolve the conflict. When that resolution is merged with other users, unless there's a conflict in the conflict-resolution, everyone sees the merge conflict go away.
Another valid CRDT operation is when a user modifies their version of the document in the region of the merge conflict prior to seeing the conflict, and broadcasts their edit. Then the merge conflict itself would update to absorb the valid change. In some cases, it might even disappear.
In principle, you can build a whole workflow stack on top of this concept, with sign-offs and reviews, just as with version control. I have no idea how well it would behave in practice.
It wouldn't make much sense to me to just plug a text CRDT in place of a standard text diff. CRDTs like automerge are capable of representing more complex tree structures however and if you squint you can sort of imagine a world where merging source code edits was done at something more like the AST level rather than as lines of text.
I've had some ugly merge conflicts that were a mix of actual code changes and formatting changes which git diffs tend not to be much help with. A system that really understood the semantic structure of the code should in theory be able to handle those a lot better.
IDEs have powerful refactoring support these days like renaming class members but source control is ignorant of those things. One can imagine a more integrated system that could understand a rename as a distinct operation and have no trouble merging a rename with an actual code change that touched some code that referenced the renamed thing in many situations. Manual review would probably still be necessary but the automated merge could get it right a much higher percentage of the time.
1. this doesn't require modifying git's merge algorithm itself; just having custom merge drivers built on top.
2. Is using more space (edit history of CRDTs vs a regular json document) worth it for automatic merging?
They utilize the same YJS[3] library mentioned in the article this thread discusses, and their GitHub repos include some useful working demonstration application code.
[1] - https://joedocs.com/
[2] - https://coronavirustechhandbook.com/
[3] - https://docs.yjs.dev/
I just don't remotely see a use case for this. Real-time human collaboration in general fails at a scale much smaller than this, and not because of the tools available.
If I was going to take an attempt at justifying the importance of CRDTs, I would say:
CRDTs are the future because they solve digital document-level conflict.
They don't bypass the problem the way that diff/patch/git conflict resolution does, by requiring human intervention.
Instead they truly and utterly obliterate the digital conflict resolution problem: a group of people editing a document can separately lose network connectivity, use different network transports, reconvene as a subgroup of the original editors... and their collective edits will always be resolved automatically by software into a deterministic document that fits within the original schema.
If viable, this has far-reaching implications, particularly related to cloud-based document and sharing systems.
That is, say you get a hundred machines editing a document. They split into partitions for a time and eventually reunite to a single one. What sort of coherent and usable data will they make? Without basically electing a leader to reject branches of the edits, sending them back to the machines rejected?
It's no doubt possible to construct application that don't behave correctly for certain combinations of edits -- but the datastructures themselves should be robust under any re-combination of the peer group's operations.
Edit / addendum: to phrase this another way and perhaps answer you more clearly: it's a responsibility of the application designer to come up with a document format for their application (and corresponding in-app edit operations) that will tend to result in 'sensible' recombinations under collaborative editing.
My sense so far is that this is the tradeoff; the complexity moves into the document format and edit operations. But that's a (largely) one-off up-front cost, and the infrastructure savings and offline/limited-connectivity collaboration support it affords continue to accrue over the lifetime of the software.
That is, this only works if you can somehow force well formed to be the same as semantically valid. Those are usually not equal.
Validity checks could include a syntax parser as a first-pass, followed by unit test as a second pass. If any of the verification steps fail, then the application can display that there is a problem with the file (and hopefully some context).
The authorities in that example are the software that checks the program; it can run locally for each participant in the session, or there could be a central instance that runs them, depending on the preferred architecture for the system.
None of the above necessarily requires discarding edits; but in some cases participants might choose to undo edits that caused problems, in order to get back to a valid document state.
Any multiplayer game does this. Git does this as well.
So of course you can do this, it's a matter of how you reconcile conflicts. Real-time interactive games will generally choose a FIFO ordering based on what came into the server's NIC first. Git makes the person pushing the merge reconcile first.
For docs, live editing seems to work the same as in games. Reconciliation for the decentralized workflow will be interesting, but it's just going to be minimizing the hit to a user when their version loses the argument.
Games, similarly, typically have a master state server. Yes, they optimistically show some state locally, but the entire process checkpoints with a central state constantly. (Else you get jerky behaviors pretty quickly as the states diverge more and more in ways that can't be reconciled without committing to a branch over another.)
Edit: that is, I would think the point is to force more, smaller arguments. Anything else puts more at risk as you lose one. Right?
The best way to prevent damage from arguments is to avoid them. Like how docs shows cursors so you know to avoid areas that are actively edited. Combined with the tiny incremental changes (often 1 char), users assume that any conflicts are due to each other instead of any distributed inconsistences.
Optimistically, I can see how they help. But, pessimistically, it looks like they just make the bad case worse.
But for a contrived example, a game with hundreds of players, backed by an enormous JSON document, where the game engine is in charge of making sure each move makes sense: A CRDT could enable that, and each player could save a snapshot of the game state as a simple text file, or save the entire history as the whole CRDT.
Or as a less contrived example, instead of a game, it's a chat client, and it provides rich text a la Matrix, but there's no server, it's all resolved with CRDTs and all data is kept client-local for each client.
There are a lot of cool things you can build with a performant CRDT.
Here is an interview with someone using CRDTs to build an edge state product that answers this question at a high level.
Has been used in RTS games to synchronize 1000s of units across low-bandwidth connections.
Input may be delayed by latency which can be mitigated with client-side prediction. Cosmic bit-shifts & indeterminism can be a challenge in longer sessions but peers can sync with eachother when there is an OOS.
I think in practice what you would do (if your use case allowed it) is use CRDTs, but periodically checkpoint and trim them when you know everyone has synced. That gives you very similar properties to the video game world and still has the features of not losing peoples edits when they make them offline.
A good read.
For those looking for more information, have a look at the information collected at http://crdt.tech/ (Disclaimer: I am involved, though Martin did the bulk load of the work.)
If you are into CRDTs for collaborative gaming, we are looking for partners and investors: https://concordant.io (Disclaimer: I am technical advisor in its team.)
https://docs.riak.com/riak/kv/latest/developing/data-types/i...
Currently we are in the "discovery of new crdts" and "engineering and implementing of older crdts reliably" phase, and in some cases "discovering when not to use crdts".
The crux of the this issue is that crdts that play nice with human expectations in regards to collaborative document editing are not known, possibly excepting automerge (yjs). As it's a 'softer' concept will no good axioms, there is no solid theory on how to combine the theoretical requirements of crdts with human expectations.
> 1. Adding wins over deleting.
yeah, so, _maybe_ you can remove elements from your set. If you're lucky. I dunno about all that...
I think it's more accurate to say that maybe you can remove elements from your set... unless another actor wants them in the set.
That's not always the behavior you want. But if it is, it's great.
All the implementations are: 1. javascript or 2. dependent on their chosen method of synchronisation or 3. incorrect.
The result of a two week long search is that I'm reimplementing the stuff myself...
I can't speak to its usability as I'm waiting on a 1.0
It does work and can actually be used as a backend for the JS implementation if you use the wasm backend we've built. In fact, this is how we have tested it, by compiling to WASM and running the JS test script against it.
My sense is that it's not a particularly flexible approach, and falters under adverse network conditions, even though it feels very tempting.
Also, any prior art on CRDTs over relational data? I suppose each single field would potentially be an "editable space", but on a long document (say a Javascript code block), updating the whole thing in the db by overwriting it with each edit would not be very efficient. Seems like there could be a datatype that was a little smarter than "text", that could handle more granular updates, that implemented a CRDT index under the hood? I'm working on a VCS for relational data [1] which is more on the not-so-realtime-collaborative-editing end of the spectrum, but would really like to figure out how to handle real-time collaborative editing in this space.
Maybe over WebRTC? I found a little bit of prior art w.r.t. CRDT over WebRTC. [2]
[1]: https://github.com/aquametalabs/aquameta/tree/master/src/pg-... [2]: https://github.com/mattkrick/rich
- An interesting case where CRDTs failed is Xi-editor, where they tried to use CRDTs as the basis for a plugin system [1,2].
- One of the biggest problems with CRDTs is the overhead needed to keep track of the full document history. The automerge [3] project has been working on efficient compression of CRDTs for JSON datatypes.
- The idea of monotonic updates is really appealing at first, but I was disappointed when I realized there's no good solution to handle deletions. Tombstones, to me, seem like kind of a hack, albeit a necessary one. Practically, CRDTs aren't the silver bullet they might seem like at first.
- Another lesson learned is that when ten people are editing the same paragraph, there's not really a right answer. I think the key to implementing CRDTs is doing it at the correct level of granularity.
- ProseMirror intentionally chose NOT to use CRDTs [4].
- Some more good references are [5,6,7]
[1] https://github.com/xi-editor/xi-editor/issues/1187#issuecomm...
[2] https://news.ycombinator.com/item?id=19886883
[3] https://github.com/automerge and https://github.com/automerge/pushpin
[4] https://marijnhaverbeke.nl/blog/collaborative-editing.html
[5] Kleppmann 2020, "CRDTs: The Hard Parts" https://www.youtube.com/watch?v=x7drE24geUw with HN Discussion: https://news.ycombinator.com/item?id=23802208
[6] Kleppmann 2019, "Interleaving Anomalies in Text Editors" https://martin.kleppmann.com/papers/interleaving-papoc19.pdf
The nice thing about CRDTs is that each individual message can be end-to-end encrypted (like WhatsApp messages), and then re-merged by all the clients locally.
A local-first database with such an encrypted sync property would would be amazing for building lots of apps with the ability to sync data between users or between your devices seamlessly. The challenge I ran into my initial experiments is that CRDTs need to be compacted/merged in various ways to stay efficient, but encryption gets in the way of that a little when considering server backups / high availability.
Following the golden rule, I always post a link to a series of papers comparing the theoretical properties of CRDTs and OT – here's the latest one:
Real Differences between OT and CRDT under a General Transformation Framework for Consistency Maintenance in Co-Editors
Proceedings of the ACM on Human-Computer Interaction 2020
Chengzheng Sun, David Sun, Agustina Ng, Weiwei Cai, Bryden Cho
It’s an evolutionary series, here’s the rest I believe: https://arxiv.org/search/cs?query=Sun%2C+Chengzheng&searchty...
My biggest issues at the time were around CORS as with a PWA you can't simply use every server the user enters, as the same-origin-policy keeps getting in your way.
Any time you see mention of "a server", you can in fact replace it with "a synchronised collection of servers acting as one".
OT involves logic on the server, not just storage, so it's not really OT as generally meant, if using S3 and a collection of peer clients only running client logic. S3 doesn't have enough interesting logic for that.
However, if you try the thought experiment of stretching "a synchronised collection of servers" to be all of the peers, no S3 even required, and then do OT with that, you can!
The result behaves exactly like OT in terms of things like document editing, conflict resolution and history garbage collection, rather than behaving like a CRDT.
It has different timing and reliability characteristics from single-server OT though. Those characteristics depend on how the virtual server is implemented on top of the synchronised peers, and especially how it implements peers coming and going.
If that sounds like OT-on-p2p has similarities to CRDT-on-p2p - they do, and they are not the same. Roughly speaking, CRDT-on-p2p has lower latency relaying updates between interested peers, because it doesn't need to globally synchronise. However with some fancy techniques you can make OT-on-p2p fast most of the time as well, and retain some of the OT benefits.
Those two behave differently but there are some common characteristics. Once you have the cluster idea, it's not out of the question to mix and match bits of OT, CRDT and other kinds of Distributed Transaction on a per-operation basis for different ops on the same data, depending on the characteristics you want them to have.
There are many trade-offs in the characteristics.
If you squint a lot, that's sort of, kind of, what realtime network games do implicitly, without a clear theory underlying it. They also add predictions and interpolations.
Why? What my app is doing is quite simple:
1. Every time the user changes something it writes the change to a journal and
2. executes the change on a local cache (to update the UI).
3. Then it starts a background sync by fetching the latest version from the server
4. executes the change on the fresh data from the server
5. uploads the transformed data with an etag to avoid overwriting parallel changes from other clients and
6. removes the change from the journal (and updates the local cache) if everything worked just fine.
So you could argue that using the etag is some kind of logic, but I think that is not what you mean with 'involves logic on the server'.
This implementation certainly doesn't work for all use-cases (e.g. high throughput/low latency), but given that it enables even offline-scenarios, I think it isn't that bad either.
Immutability makes data control way harder ...
IMO, when looking for predictions of the future, listen to the ones that have a track record of accurate predictions.
To some extent, I see the value in "don't listen to predictions from people with a record of inaccurate predictions".
On the other hand, people who make a lot of inaccurate predictions are, at least, taking a chance by saying out loud what they are thinking. They know their predictions might be wrong, and hopefully they learn and update their methods along the way. It is a nice way to grow.
Back in my Austin road cycling days, there was an amazingly talented junior rider who had a reputation for crashing hard and often. Some of us wondered how many pieces his frame could be split into. He survived the crashes, best I can tell. I'm pretty sure his confidence, resilience, and bike handling skills got better with every crash. Pushing the limits and being wrong in these kinds of situations may not be everyone's style, but it can work.
Don't be sad. What you did was a huge inspiration to me and I think a lot of people. It showed the possibilities even if it wasn't "optimal".
Commenting here in hope you'll see it. I skim to read most things, I only completed reading this article because I noticed 'I worked on Google Wave'.
As the author explains, OT relies on some ordering of system events, and CRDTs don't. That means CRDTs need to be commutative (and probably associative), and OT doesn't.
So, OT is less scalable but more powerful, and CRDTs are more scalable but less powerful (in theory).
It's sort of like comparing Paxos/Raft to Bittorrent.
(I am not an expert on OT.)
What we need is a community set of peer relay nodes, on top of which data structures can be synced. Infra companies are well setup to provide this firebase like store layer on top, but for any generic data structures (lists, dicts, array, etc)
With this, any saas application is at a disadvantage, because data is no longer tied to the application!
Actually, CRDT is not a single data structure or even algorithm. It is a term for several families of data structures and different algorithms on them. If your task is not editing text, you may find a simple and already implemented CRDT for your case.
Try Y.js: https://github.com/yjs/yjs
Amen, brother.
So, sort of like email ?
https://www.inkandswitch.com/local-first.html
Referenced as a reason why CRDTs are a good way to enable this.
Don't write like this. Respect your readers and help them comprehend. Expand acronyms as early as you can, ideally at the first mention.
Mods, can we expand the acronym in the title of this submission please?
I mean, if SLR's became DSLR's... I just assume any "D" means we're digital now! :)
It's called the link. All an author has to do is link the first instance of an acronym or piece of jargon to some authoritative description, and you get the best of both worlds: readers familiar with e.g. CRDTs[0] can just keep reading, and the rest can click the link and find out.
[0]: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
It is especially useful when writing a technical document that utilizes multiple products/stacks/terms. Creating links to quality sources for those items gives someone new to the content a good source to go deeper into those pieces while allowing me to focus the article on the specific aspect I'm writing about.
Conflict-free Replicated Data Types (<a href="...">CRDT</a>)
A lot of people seem to be questioning why you'd need this when you could just provide a full link. Personally I read a lot of technical documentation and having acronyms written out in full would almost always be enough. Otherwise the whole document is likely over my head, or it's just a bad acronym.
Hypertext is great, links are practically free, I encourage authors to be liberal in applying them.
The same arguments for and against using libraries apply here, and it's up to the author which works best for their piece.
Just like libraries, sometimes it is and sometimes it isn't the best approach.
For example, in a "How to do $BASIC_THING in python" article, putting an intro of "This is what a variable is" may not be a bad idea. Meanwhile, in a "Writing an operating system from scratch in an esolang I wrote" article, maybe you'd be better off linking to previous blog posts or other resources.
Obviously these are both extreme examples, but I think it's still a valid view.
Anyway, blog author here - sorry I didn't explain CRDTs earlier in the piece. It didn't occur to me that people would be confused.
It never ceases to amaze me how many websites for restaurants or whatever neglect to mention basic things like what state (and country) they're in. Even newspaper web sites assume that we know that the "Chronicle" or the "Ledger" or whatever generic name is the local paper for East Bumblefuck.
Which they can easily provide themselves.
> If everything is written with jargon and abbreviations with no context [...]
It is not, for the intended audience.
The video also gives good context for the article, even for a beginner to the topic.
Spelling out "Conflict-free replicated data type" doesn't really help beginners all that much and non-beginners will just use "CRDT" anyways.
We don't need every article about the web to spell out HTTP right? I don't get why the author is getting beat up just because his free content isn't convenient enough.
I read the title, wondered what CRDT was, and started reading. In the back of my mind I was wondering what CRDT was, but reading the article felt like I was going on a journey. Every term that needed to be defined was defined. Finally, when CRDT was mentioned in the article, it was immediately defined.
I generally agree that throwing acronyms around without defining them is not fair to the reader, but I don't think this article did that at all.
Whenever I see this writing style, such that I cannot find a thesis in the first two paragraphs, I almost universally discard the writing as a waste of time.
> I've spent the last decade working on OT, and have always thought it was the right way to implement a collaborative editor. Then something amazing happened.
Instead, we get this:
> I saw Martin Kleppmann’s talk a few weeks ago about CRDTs, and I felt a deep sense of despair. Maybe all the work I’ve been doing for the past decade won’t be part of the future after all, because Martin’s work on CRDTs will supersede it. Its really good.
That seems like the opposite of burying the lede. The main point of the story is _not_ that CRDT stands for Conflict-free Replicated Data Type, it's that the author now favors CRDTs over OT for collaborative editors.
That can be seen by glancing at the comments on this page.
Engineers, when talking about technical concepts with acronyms, always expand them for the first time to your readers!
I’m a little embarrassed to admit I didn’t even notice.
I, for one, am grateful that you took the time to write it.
Anyway, I've updated the introductory paragraph to make it more clear.
In contrast, I like way more a different approach on explaining (mostly see it on Cyrillic forums) -- instead of guiding you by hand, they just give you clues where to look for. That way, knowledge givers are way more approachable, because it costs them very little to chat back something like "look for CRDT", than go into in-depth explaining. In the end -- there's way more information, and from top experts in the fields.
This is way over the top.
I thought the author did an amazing job of discussing a highly technical topic in a very approachable way. Every blog on HN should aspire to write like this! It was so good it got me reading other posts even.
Yes, it would have been nice for us non- domain experts if the author had done the classic "Conflict-free replicated data type (CRDT)" thing, but you can easily just say that, ya know? "Hey, it would be helpful if you expanded CRDT early on."
I found the article quite good, and if you had genuinely been motivated to engage with the content you could have highlighted the acronym and searched for it. There is a wealth of good info for "CRDTs" that comes up on the first page of Google, Bing or DDG.
Does the acronym actually illuminate what they are or how they function? I submit to you that it probably doesn't.
The most recent guideline we added says: "Please don't complain about website formatting, back-button breakage, and similar annoyances. They're too common to be interesting." I suppose that complaints about writing style fall under the same umbrella.
Not that these things don't matter (when helping people with their pieces for HN I always tell them to define jargon at point of introduction), but they matter much less than the overall specific topic and much less than the attention they end up getting. So they're basically like weeds that grow and choke out the flowers.
(This is not a personal criticism—of course you didn't mean to have this effect.)
https://news.ycombinator.com/newsguidelines.html
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...