Hacker News Reader

Comment by zemo | Hacker News Reader

I use CRDTs in production at Jackbox for audience functionality and honestly I don't know why the only thing that people talk about when it comes to CRDTs is collaborative text editing. Like, sure, cool, but that's literally a single problem domain and like, Google Docs already exists and works well for the majority of users so how many developers actually need to create a collaborative text editor? CRDTs are an incredibly abstract and versatile concept; collaborative text editing is a tiny problem domain. I would really like to see more writing about CRDTs that is NOT about collaborative text editing.

View on HN

Author of the blog post here. I’ll let you in on a dirty secret: I agree with you.

I see text editing as the beachhead for this tech. Text editing is hard enough that other systems don’t work well, so you’re kind of forced to use OT or CRDTs to make it work. And text documents are lists of characters - so once you’ve made it work there you have an implementation that can also sync arbitrary lists. At that point, editing maps (js objects) and adding support for arbitrary moves and reparenting will allow us to add real-time editing to a lot more software.

I think there’s a lot of interesting software architectures that can be enabled by making everything in a system a CRDT document. But text is still the beachhead. And a good litmus test for performance. And an important application in its own right.

on the one hand, the generality of the text editing solutions is really powerful, and I see what you mean in terms of that solution generalizing to other domains. But on the other hand, I always think back to how popular Memcache or Redis were even early on when they had very very few features. Just having a fast, in memory cache of strings empowered a lot of interesting new product development. I really wonder how much the average developer on a random project could get out of an appliance that lets you create and mutate an arbitrarily large number of values of the well-known, simple CRDT types like g-counters, pn-counters, 2P-sets, etc. Most of the literature is focused on "how do we merge the most complex data type", and not questions like "how do we manage the entire lifecycle of CRDT stores", "how does a CRDT store fit into most people's stack", "should CRDT types be added to existing stores or should developers expect dedicated CRDT-only stores", or "do people generally need to define their own CRDT types or do most people just want a box full of common ones". I hand-rolled my own CRDT setup just to get the most simple CRDT types because I didn't see anything out there that makes directly consuming CRDT types by application developers accessible. E.g., you make a g-counter and literally all a client can do with it is increment it or read its value. That's it. We have that, and it's totally useful! We also do entirely state-based replication because expressing the operations on the data would be so much larger than the data itself. But our situation is just so off-base for many people because clients are only ever interested in the current state (and never care about the past state), and we can safely just ignore the problem of when to delete the data; we just keep it around until you're finished playing the game, and then delete it when the game is over.

Etcd does some of this stuff, but in general it sounds like you have a useful opensource tool inside you that wants to be shared with the world. I can't wait to see it made.

How could CRDTs be used to collaborate in a project made with a visual programming language that consists of interconnected operators? Is it necessary to serialize this graph?

Here’s one written by an employee of Redis Labs about CRDTs for georeplication of databases for strong eventual consistency. [1]

There's also 2-year old list of CRDTs used in prod [2].

I know Azure CosmosDB (NoSQL) advertises it's own use of CRDTs for global replication.

[1] https://www.infoworld.com/article/3305321/when-to-use-a-crdt...

[2] https://github.com/ipfs/notes/issues/404

I love the Jackbox games.

However in my experience most games's shared world state cannot be reconciled if one player's action is inserted in time prior to another player action. Or future actions reveal hidden information.

Even turn based games have turn timers. Including Jackbox games!

Among public databases there may only be two correct CRDT implementations in the world - Cassandra's and etcd's. Compare to how many database products make CRDT or CRDT-like promises, and how few probably actually work.

Lots of software promises CRDTs-like-over-actors, e.g. Erlang, the new Cloudflare product. Not really qualified to talk about them.

Most applications normal people use boil down to chat conversations - chat apps, Gmail, Facebook, collaborative text like you said, all sorts of log-like things - or serial, recommendation-driven consumption - YouTube, Netflix, which use incremental / batched cells in a matrix to update their recs. This stuff is insensitive to order and time. Then there's e-commerce, which is inherently linear, centralized, single-actor, all-or-nothing transactions, CRDTs provide no advantages here.

It's tricky. On the one hand you'd get this interesting discussion of CRDT applications. On the other it may wither under intense scrutiny.

Would you consider being the writer of one of those articles?

well I really walked into that question huh

I'd read this article.

Or, if you'd rather not, can you share some examples of problem domains, that we at home can search for?

a lot of other use cases are interested in using CRDTs for complex mutations with a relatively small number of users. Most collaborative editing sessions will be a small handful of users. Our use cases are things more like "three thousand people are presented with a multiple-choice question simultaneously and answer it simultaneously and we have to show the results on an xbox and the entire thing has to happen in under 15 seconds". We mostly just use the standard g-counter and pn-counter types. We have a thing that's like ... a high score list for a trivia game that doesn't have a well known type (it's kinda like ... having a zset where only the top N items in the zset are replicated) and a CRDT implementation of a ring buffer of strings for audience comments in ... Quiplash 2, iirc. I'm a bit tied up since we have a new pack coming out October 15th, but I might have time to write some stuff up after that.

If you write an article and post it here I will read it.

Followed you in case you ever do! I'm a big fan of Jackbox and I'd love to read about where CRDTs fit in your stack.

+1 Nicely done.

We want something like that for distributed collaboration in Ardour, a cross-platform DAW. The relevant state is serialized as an XML file, and used natively as a complex object tree. Users want to be able to edit (in the DAW) locally, then share (and merge) their results with collaborators.

I've plugged this collaboration project a few times recently, and have no relationship to it other than discovering it (via YJS' "who is using" list[1]) and finding it fascinating:

http://cattaz.io/

What I find most interesting about it is that it has reduced the state of multiple 'smart' user-facing widgets/apps into a common, lowest-common-denominator format (a text document) that lends itself more easily and intuitively to collaborative editing and CRDT operations.

I don't know for sure whether this is the path forward for CRDT-based applications in general, but I think there are valuable ideas there. It does raise the possibility of the widgets/applications occasionally being in 'invalid' states; but rarely in a way that the human participants wouldn't notice or be able to fix themselves.

Whether that scales to the complexity of the state management for a multi-track audio editing session, I don't know; but it could be instructional to compare.

[1] - https://github.com/yjs/yjs#who-is-using-yjs

in real-time? Well, I have thoughts, but I'm not super familiar with Ardour itself, so I'm not sure if you're trying to merge during a live performance or if you're talking more of a distributed studio recording session type situation. I have working knowledge of Reason and Logic and ChucK (which I use with JACK and do some networked OSC stuff with, although I haven't touched it in a few years).

The approach we use at Jackbox for making the state of an xbox game mutable to thousands of live viewers on twitch is to have lots of little CRDT values, mostly just counters and sets of strings, and you merge the little values independently of one another, which is very different from the situation of editing a text document, which is typically structured as one big value. I wonder if, for a DAW, you could merge at the track or control level instead of the workspace level. E.g., communicate as an independent value the state of an individual fader, and communicate either states or operations on that fader and have each client merge them. In this example, the fader's state would be encoded as a PN-counter with a range condition, and you'd replicate increment and decrement options, like it was a networked rotary encoder. So every mutable thing in the DAW would be a value having operations that can be merged, instead of having a single big value representing the entire state of the DAW. My use-case is also funky because I have potentially thousands of writers writing to the same key, but only a single reader, and the reader doesn't need an update at every change, so I use state-based CRDTs, but I think most other people using CRDTs use operation-based CRDTs. Also not sure how you would mutate two separate values transactionally or if that's a thing you even need.

Not realtime. Users would sync periodically during their working process.

There are lots of mutable things in a DAW that are not numeric parameters.

The state of a playlist (just think "some time ordered list of objects") is not treatable in the same way as the value of a fader.

If you had a context-aware XML parser and access to timestamps for every XML node, you could do the human-aided merge by considering each node and just using the latest version of the same node, falling back to the human when there's a deletion conflict (for example). But this doesn't actually merge the attributes of a node or deal with conflicts between otherwise sequential edits.

We use CRDTs for having a scalable, availability first, service discovery implementation. I’m sure there are more uses out there. We use Akka Distributed Data and there are many users of that.

just wanted to say, i really like Jackbox's games and its really fun to see you here! i've been exposed to your games via youtube vids and it feels weird (but cool!) to have that intersect with HN. would love to read more about how you use CRDTs to make it all work! (i'm pretty clueless re: multiplayer games so even a high-level explanation would be interesting)

Holy shit CRDTs totally make sense for Jackbox. (Btw thank you for the awesome games!)

Thanks for your work on Jackbox! Never had an issue in the games and they are perfect for parties. I always thought it would be fun to make a Jackbox-type game system.

Why not writing an article about how you use them?

1000% this.

Reply on news.ycombinator.com