Building Document-Centric, CRDT-Native Editors
blocksuite.io
blocksuite.io
I remember reading a computing magazine in the early 1990s that promised a future where we would decompose applications and the OS would only really worry about a file, and you would bring functionality to the file.
You would in essence be able to build your own perfect word processing environment (for example), by bringing Company X's editing tools, Company Y's grammar checking and spelling tools, perhaps some embedded spreadsheet tables from Company Z if you were writing business reports, and so on.
We kind of have this a little today with browser extensions, in that we can extend functionality onto a webpage we're viewing, but our environments are still very application-centric and not workflow or content-centric at all.
This article shows an application that _might_ be interesting (and the CRDT is a mandatory requirement in today's environment), but while the OSes we use require us to do this sort of work in a windowed application, it won't quite appeal to me as having the full potential.
I often think back to that article as it made me quite excited about the future of user interfaces and how operating systems could support workflows tailored to the individual and the task they wanted to achieve. This was all in a time when we had moderately novel ideas in OSes popping up (Windows NT, OS/2 Warp, NeXT, etc.), and just before the web was starting to get popular.
No business model means no long-term maintenance, no polish, no marketing, no attention to detail, no usability iterations, and so on. Quality software that is easy to use (especially for non-technical users) is extremely expensive. Developers are expensive. So instead you get "WIMPs" (Weakly Interacting Massive Programs) that win in the marketplace because they are more polished, more maintained, and more supported. (Because people pay for them.) Lately most of these WIMPs are in the cloud.
The more I've matured as a developer the more I've realized that business models are the tail that wags the dog and that a lot of the landscape of software is defined by what people will pay for and/or what is structured so as to make people pay for it.
Business models are also why things are increasingly centralized and in the cloud. It's not because it's inherently better, though it does make certain things easier to implement. In many cases it's a lot worse: higher latency, not available offline, much more limited and slower UI, etc. It's because the cloud is DRM and makes it easy to force a subscription. I mean look at Figma... there is no fundamental reason that had to be in the cloud except that it provides a ready-made subscription model that users can't evade.
Or maybe we have done it in the past(interactive software in SmallTalk), but have forgot about it.
Also, the business reasons are not prohibitive - if lot of users use the workflow model, there can be a store where they can request, raise funds for plugins with a specific functionality. Developers won't ignore it, even if the moat is weak, as there is potential revenue. It would be like consultants providing solutions rather than selling a product.
I mean look at this stuff you could do on a machine with 256KiB of RAM and an 8086: https://www.youtube.com/watch?v=KMUT9TEoe4Y&pp=ygUYbXMtZG9zI...
All of this was completely abandoned and forgotten because there's no money in it. Make software like that and there's no moat, and make it local and people will pirate it. Lock it down in the cloud and lock down the data and people will pay you.
That which gets funded gets built. We get shit because we pay for shit. People won't pay for good software because the flexibility and user-centrism of good software allows them not to.
Nevertheless, I am more optimistic about such a product. The key issue is building composable interfaces to which plugins can glue in.
Piracy is not such a big problem as one can give the product free of charge to users and charge for hosting or charge commercial businesses for whom piracy is not a risk worth taking.
If anything, the business implication of a ubiquitous workflow interaction is its threat to the ads model (users dont visit social media sites but read in data via apis). The ad model might need to be replaced by something like micropayments to content creators, but that is a distant issue.
The only problem is that it’s read-only. If QuickLook extensions could provide write capabilities too, you’d have 90% of a document centric setup.
Tree-sitter, LSP, various linters -- we're getting there!
I just recently put into production a large scale document centric CRDT editor using yjs and Vue3/Vuetify. This is a platform agnostic WYSIWYG editor that supports around 30M daily users. It was a really intense experience writing this software! It uses YJS as the document, then creates an event sourced change stream (ydoc changes) that is streamed in via a websocket (APIGWV2) lambda into a dynamodb, then projected into S3 as plain old JSON for service load. The editor presents "what the web sees" for platform preview and is interpreted from the ydoc. All serverless. Multi user is awesome! History is awesome! The whole thing is just really cool!
I ran into two major issues: upstream updates being too large for a websocket which I fixed with signed URLs to a S3 YJS triggered lambda that inserted these very large events - which are exceptionally rare, like when I user copies/pastes a giant doc. That and multi user hierarchical updates, like where a user updates a parent node and that triggers an update of the children messing up the other user. The trick was to flatten the hierarchy into a linked list on save and save the doc as a flat ymap structure in the DB. I had to chunk serialized strings into dynamodb because of the ~400k row limit, but no regrets, loads are very fast over WS even when chunked. Lots of learning. Really fun system to write.
What I expected instead: a way to consider the document as the first-class citizen in people's workflows (especially local-first flows, since CRDT is mentioned), which is a big challenge considering that all filesystems are still file-oriented (ie. if you want two versions of a document, you end up with two files, which then become two documents since there's nothing binding them).
What I'm still not sure about is how superior CRDTs are if you're building a centralized service. CRDTs can be great for offline mode or syncing data across long time periods, and not necessarily for an online and realtime multiplayer experience. I know they work just fine in such a situation, but they do come with additional cost in logical complexity, memory, and compute.
Does anyone have experience with doing first "old school" sequential data patching (which would need some additional work to support undo and synchronization with clients that are trying to make simultaneous changes) and then switching to CRDT? Last time I tried it was too costly in every way, so I'm curious.
I think it’s a big selling point.
I recently built a centralized, multiplayer, editor as a POC.
I did one iteration with hand rolled operational transforms (what google docs uses) and one with yjs.
Yjs just kind of works, where OTs require me to handle different kinds of data in bespoke ways and resolving conflicts is much more difficult due to the manual, one off nature of OT based systems.
Yjs’s CRDT approach is much simpler to get going with and, imo, CRDTs are easier to reason about than OTs.
Yjs’s implementation of CRDTs is quite a bit more space efficient than other implementations I’ve seen.
I haven’t dealt with either in a production scenario yet, but I’m going to be moving forward with Yjs crdts. There are native bindings as well.
> OTs require me to handle different kinds of data in bespoke ways
I'll admit I haven't tried to implement a general OT solution, but CRDTs are also extremely complex functions on the base building blocks of your data. I have tried Yjs and it does indeed hide away a lot of this complexity for you, but I would be surprised if there isn't an equivalent OT library (a quick Google turns up https://www.npmjs.com/package/@otjs/state-machine). Furthermore the CRDTs uses quite a lot of memory as your project scales up, so I'd keep an eye on that.
I'd be curious to see more of your project even if it's still in progress, is there already something live?
I can’t really speak to CRDTs more generally, but yjs provides a few fairly general data structures which “just work.”
In my (non extensive) experience with OTs, you need to write bespoke, application specific conflict resolution, but I haven’t tried that OTJS library.
> I'd be curious to see more of your project even if it's still in progress, is there already something live?
Not yet, hoping to have something by the end of the year. I have the domain name “outlines.work” if you feel like checking it out later.
The yjs parts are very similar to this example repo, though:
There's no splitting or isolation available here, you need to load the entire Y.Doc to get any of the content of the page. The content of the Y.Doc is opaque bytes, and if you dump it to some other representation (like JSON) then you lose the internal Y.Doc counters that make conflict resolution work.
https://docs.yjs.dev/api/subdocuments
If you compare Ydoc for a very large page with many items to a web API maybe it seems like it could be a bit big, but traditional file document in an editor like Pages are frequently several megabytes, it really doesn’t seem spooky to me in a truly local first app where you will delta sync.
But yeah …