Learn Yjs Interactively | Hacker News Reader

Learn Yjs Interactively

learn.yjs.dev

309 pointspaulgb42 comments

Hey HN! I'm the developer at Jamsocket who made this. In case you're not familiar with Yjs, it's a CRDT library for building collaborative and local-first apps.

The thing is, if you're not used to working with distributed state there's definitely a learning curve. Even simple things like incrementing a counter — the "hello world" of JavaScript framework demos — get tricky when dealing with multiple clients. Worse, a lot of tutorials are just like "install this library and text editor integration and boom you have an app", which doesn't give you a good mental model for what's actually happening.

So we made Learn Yjs! It's an interactive Yjs tutorial. I wanted it to be really intuitive for people just getting their feet wet with local-first development, so there are lots of explorable demos and coding exercises. The idea is to use interactive examples to build an understanding from the ground up.

Hope you like it :)

Woah! Beautifully done! I wonder if I will finally understand CRDT with this. Will go through it over the weekend.

Congratulations!

I am just looking at this but quick question. Is there an example for building a local-first app? Also, I working with plain vanilla HTML & JS and Alpine. Can this be used in my apps?

Y-sweet (which this is built with) allows you to make a Yjs app local-first by passing “offlineSupport” to the provider when constructing.

There’s a demo of this here: https://demos.y-sweet.dev/color-grid

The source for that demo is here: https://github.com/jamsocket/y-sweet/blob/5fa6941cf5568f4a3f...

That demo uses react, but there is a vanilla js version of the same demo here: https://github.com/jamsocket/y-sweet/tree/main/examples/vani...

(The vanilla js version is not local first, but it would just be a matter of passing the same offlineSupport flag to make it so.)

Thank you. This is the direction I needed. Will try out

I just noticed that if you install the example service after setting up an account, it installs a version of @y-sweet/react which doesn't offer the offlineSupport prop.

The version specified is ^0.5.1. After switching to 0.7.1 as the repo you linked uses, all is well. Probably not a big deal for most newcomers, but it might be worth updating dependencies in the example service.

Thanks Steve, good catch!

Thank you. Keep going. Very good topic to expand

> "install this library and text editor integration and boom you have an app", which doesn't give you a good mental model for what's actually happening.

I was interested in working with CRDTs and collaboration in general for a while now and this has been the biggest issue whenever I tried to get into it. Websockets also seem a bit harder to find (free) hosting for.

Btw, your other tutorials on CRDTs were also a great help.

Cloudflare workers can host soketi which is websocket.

Thank you!

As far as websocket hosting, there are a few services out there. We offer a pretty generous free tier for Y-Sweet [1] with 10GB of storage. There's also PartyKit and Liveblocks, though I'm not sure what their pricing is like.

[1] https://jamsocket.com/y-sweet#y-sweet-pricing

Does this work with Yrs [0] or is that a separate project? Also, does Y-Sweet work with any CRDT library like Automerge or Loro or only Yjs? I ask because they all seem somewhat similar in terms of being binary files being sent back and forth so I was just curious how compatible Y-Sweet is with other similar CRDT implementations.

[0] https://github.com/y-crdt/y-crdt

Yjs is nice. Easy to get p2p results.

Yjs backend, its persistence, eventual conflict resolution (one of editors stayed too long offline), history rewind seems like a tough engineering challenge to crack.

Always wanted to have a nice experience with block editor like Platejs https://platejs.org/

There are nice attempts simplify dev experience with solutions like Liveblocks https://liveblocks.io/text-editor However most options come at the expense of controlling your data.

CRDT keyword search on HN bring consistently good results and interest in the topic, but no good options on open source backend side. Checked it many times in the past. Please correct me if I missed something.

https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...

Most people expectations it should work at least as in Google Docs or Notion, and on the dev side it should be store privately in Postgres JSONB like format without going deep into details.

I agree. In theory you need to just distribute serialized patches but in a real world backend scenario you may need to integrate with knowledge of current document state, user identity, and possibly even some level of access control.

I’ve wanted to use Y.js with a Go backend multiple times but gave up each time due to time constraints as it’s hard to find simple reference implementations.

I’ve been checking back over the years but it still seems hard to do this outside of Node.js.

Same experience. Go is my default choice for backend too

It can easily be a full time job

You need to mirror all logic and encoding/decoding part

https://github.com/yjs/yjs/blob/main/src/utils/encoding.js

https://github.com/yjs/yjs/commits/main/src/utils/encoding.j...

I was thinking about spining off nodejs instance just for persistence and syncing with postgres

Author here — I think our own almost server is almost exactly what you’re looking for? Y-Sweet [1] is open source, written in Rust and persists documents to S3. We offer a hosted version if you want to quickly check it out, which you can use with your own S3 bucket if you want to self host eventually.

This is the first I’ve heard of Platejs, but we do have a tutorial on integrating a block editor using BlockNote [2]

[1] https://github.com/jamsocket/y-sweet

[2] https://docs.jamsocket.com/y-sweet/tutorials/blocknote

Fyi, building a collaborative editor right now and it's still hard to do outside of Node

Still better time spend than doing leetcode.

It's my go-to argument against leetcode style interviews.

I would rather ask a candidate to spend 30 mins and do research together on collaborative editing, or visualizing distances used in pgvector or similar vector database.

Imagine how far the whole colabarative editing space moved forward if 1% of leetcode grinding were rerouted

I'm working on an Obsidian plugin called Relay [0] that makes Obsidian real-time collaborative. Users can share specific subsets of their vault with different groups of people. Our goal is to make Obsidian suitable for corporate use.

We use y-sweet for real-time collaboration backends (thanks paulgb and team!) and handle authentication via Pocketbase. We host everything on fly.io.

I believe there's a lot of interesting space between "pure" local-first software and being a data serf to modern SaaS applications.

On the technical side, it makes sense to have most of your graph be server-managed but locally mirrored. We maintain a store of all user-accessible metadata on the user's device while keeping it up-to-date with SSE – similar to a simplified version of Replicache. Pocketbase makes this very convenient to work with. We modified Pocketbase to use h2c, enabling end-to-end HTTP/2 through fly's edge proxies. This allows us to serve numerous SSE connections for collections related to authentication, authorization, file sync, and other metadata.

We aim to keep all documents on the user's machine (following Obsidian's file-over-app principle). This means the Obsidian client needs to sync with files on disk that lack CRDT context when the codemirror editor is closed. We've developed novel solutions using a combination of diff-match-patch and a diff review tool that we surface to users in these scenarios.

One cool thing about using y-sweet's in your architecture (eg. separate collaboration backends), is that your customers could self-host the collaboration server, preventing you from accessing their data. This business-friendly version of partial-self-hosting (BYOC?) creates nicely aligned incentives for a more ethical kind of SaaS company -- you can build pretty much everything open source except the auth server, and then charge for authn/authz in order to fund development of the project, and the customer gets sovereignty over their data.

One interesting feature on fly.io is the `fly-replay` header, which enables dynamic request routing of websocket connections to machines [1]. This is not a redirect that the user can see, rather this allows the edge proxy to replay the connection to another host transparently to the user. Combined with fly.io's auto start/stop functionality [2], you can build something similar to jamsocket/plane with minimal code and some nice properties like controlling geographic placement and having a single URL for all document servers.

It is a super exciting space to build in right now. I'm grateful for yjs and to the jamsocket team for everything they've built (and open-sourced!).

[0] https://system3.md/relay [1] https://fly.io/docs/networking/dynamic-request-routing/ [2] https://fly.io/docs/launch/autostop-autostart/

I used Yjs in a small side project a while ago. The client side was fairly easy to learn and use. On the server side though, I wanted to deploy a single binary, but examples in languages other than Node (e.g. for the Rust port Yrs) seemed close to nonexistent, so I ended up slightly adapting https://github.com/yjs/y-websocket (Node) with persistence based on LevelDB. That's suboptimal for me. I wonder if there are good resources for implementing the server-side with persistence in, say, Rust.

Btw, IIRC existing resources for the client side are heavily focused on text editor integrations, so this is really helpful. I had to fumble a bit myself because I wasn't using a text editor integration.

You might like Y-Sweet[1], an open-source Rust server that powers the banner on Learn Yjs. It uses S3 or the local filesystem for storage, and it compiles down to a musl-linked binary that weighs a few megabytes.

https://github.com/jamsocket/y-sweet

Thanks, I'll look into it. It's pretty low on the list of search results for Yjs server, I somehow missed it the last time.

> The trick is a technique called fractional indexing, in which indices are fractions rather than integers.

How many times you can use fractional indexing before it stops working?

Sorry, just did that test - it is 1075, not massive, I would risk saying that may be major issue of this lib.

Here is the code:

let left = 0, right = 1

let rightPrevious = 2

let counter = 0;

while (right > left && right < rightPrevious) {

    rightPrevious = right;

    right /= 2;

    counter++

}

console.log(counter)

Did look very briefly into repo, but didn't find a place where it is handled and don't have time to dig it, but if that issue is true I may help with resolving it. LMK

Good point. Usually fractional indexing is done with arbitrary-precision numbers to avoid exactly this issue. We landed on using JS floats to teach the technique because it makes it more intuitive to understand what’s going on, but you make a good point and I’ll make sure we add that caveat and link out to an arbitrary-precision fractional index library.

I'm currently using Yjs to build an app that needs to work offline. It's not actually a real-time or collaborative app. However, if you think of the server as being one collaborator and the app being another, then it's easy to imagine use cases for it.

The thing I never got past with yjs was that the generalization of the todo app item-moving thing in this tutorial gets messy so quickly. If you have multiple objects that relate to each other, then you want them in the same Document, and keeping the relationships in sync has lots of gotchas (like needing the fractional indexing in the tutorial).

But then you accumulate a database worth of items and the simple toy model of putting everything into one document amounts to having to ship the whole DB (and it's history) to anything that wants to read any items. So you need to start breaking the items into separate objects and persisting them yourself, which has no documentation except for this tutorial [0] (spoiler alert: it's just an empty "Work in Progress - come back later" and has said that for years and years and years).

So you try to build a Provider to manage things, and the documentation story is the same [1].

Then you find out about Subdocuments and get happy that there's a solution to this common problem [2], but then learn that some Providers can't handle any relationships between subdocuments, and some Providers don't even support subdocuments, which goes back to issue of DIY providers not being documented.

So then you try to read some code to learn about it (after all, it's open source, we can't expect the dev to do all the work for us for free; let's do some work too!) and realize that what yjs really is at its core is a protocol. And that protocol -- the very essence of yjs -- is also totally undocumented and spread across a bunch of relatively idiosyncratic code. Unless you count a paper that documents how a yjs precursor worked a decade ago.

I didn't intend this to turn into a rant, but I've spent so much time trying to get it working over the years, and the gap between the toy 'Look it's magic!' demos and anything real is just so wide and the documentation for that part has been empty except "Work in Progress" for a decade.

I'm hopeful that some of the companies trying to monetize yjs, like the author of this post, are willing to step into that gap (which is who I'm writing this for, btw).

[0] https://docs.yjs.dev/tutorials/persisting-the-document-to-a-...

[1] https://docs.yjs.dev/tutorials/creating-a-custom-provider

[2] https://docs.yjs.dev/api/subdocuments

[3] I was really hoping that as I wrote this, the situation has changed, but it looks like all the documentation is just as WIP as five years ago and same for subdocument support.

So perplexing - in the interactive demos, the latency slider acts not as a network latency but more a debounce buffer? Why? Why? I just don't get it

We wanted to go with the simplest model of latency that would suffice to demonstrate the problems that come up when dealing with state mutated in multiple locations. The advantage of this approach is that we can do it without hiding any state from the reader — the pie chart exposes the only internal state that exists. It’s not meant to be an accurate network simulation, just to build intuition.

The interactive demos are beautiful! Is there a library you used to build it?

Thank you! The site as a whole is built with Astro and the demos are “islands of interactivity” built with React.

The demos are really running Yjs under the hood — each “client” has its own document, and when the user clicks I set a timeout to simulate the latency and then manually merge the documents into each other. (There’s a third document too for the timeline in the center.)

Other than that, I’m using dnd kit [1] for the drag and drop functionality in the todos demo, Motion [2] for the animations and CodeMirror [3] for the text editor.

[1] https://dndkit.com

[2] https://motion.dev

[3] https://codemirror.net

Thank you Jake! I just played with the motion website and loved it. I am going to try to use it sometime

Great work Jake! I love the Astro and the “islands of interactivity”. Is it possible to see the repo, if it's public? I couldn't find it anywhere.

The banner image game is silly but fun

Reminds me of the fun reddit used to have with places and such:

Team purple ftw: https://imgur.com/a/Oxc6jUA

I finally got a four leaf clover and somebody ruined it after like 2m.

Edit: Wait, where did everybody go? I think we broke it, or maybe I got banned. Kudos to the creator just the same. Neat stuff.

You definitely aren’t banned! Maybe people just tapered off for a bit — I see a bunch of cursors now.

There are two more secret emojis, by the way :)

is Yjs a platform like Convex? are they competing? would love a comparison page potentially! :pray:

You can think of Yjs as a protocol for data synchronization. It gives you a way to describe a JSON-like data structure (i.e. nested lists and maps), and keep them in sync across multiple devices.

Yjs itself doesn't provide a platform, but it's an open protocol so there there are service providers (like ourselves[1]) that offer Yjs backends as a service (other notable providers are TipTap/Hocusocus and Liveblocks).

[1] https://jamsocket.com/y-sweet

what's the main differences in your opinion between y-sweet and Hocuspocus?

The biggest difference is that Y-Sweet is built around object storage (i.e. S3 and friends) rather than a database.

This allows writes to scale horizontally without a central database being the bottleneck. It’s also a much cheaper way to store lots of documents accumulating over time on S3 than in a database. This is because compute scales with how much data you actually access, not how much data you store.

I wrote about why object storage is a good fit for this here: https://digest.browsertech.com/archive/browsertech-digest-fi....

another banger from the jamsocket team

Reply on news.ycombinator.com