Local-First Software (2019) | Hacker News Reader

Local-First Software (2019)

inkandswitch.com

185 pointsbrodo29 comments

While it seems natural to assume CRDTs as foundational tech for local-first softwares, do keep in mind designing CRDTs for your app's state may not be an easy job.

It's easy if all your app does is append to lists or keep a counter or deal with generic key/value pairs. Designing CRDTs for these cases is super easy.

But the moment you have a semantic JSON that has a grammar to its structure, you are going to need your own conflict resolution algorithms that keeps the semantic structure of your state clean and valid. That requires a lot more than slapping in a Yjs integration or Automerge integration.

Yes! I have long felt like there are CRDTs possible that maintain complex application invariants (that is, have transactions).

But when I sat down to think about it I realized that CRDTs make the problem far harder than it needs to be for today's software. CRDTs are designed for when there is no central authority, where all nodes are peers. But most software is not peer-to-peer, it's client/server and does in fact have a central authority -- the server.

This is how https://replicache.dev/ started. It's a CRDT inspired system, but takes advantage of the server to provide much stronger guarantees. Replicache mutations are transactional and enable enforcing arbitrary application invariants.

For example enforcing "only one meeting can be booked in each room at a time" is trivial in Replicache. You can just say:

  async function createReservation(tx, reservationID, roomID, time) {
    const existingReservation = await findReservationByTimeAndRoom(tx, time, roomID);
    const res = {
      success: existingReservation == null,
      roomID,
      time,
    };
    await tx.put(reservationID, res);
  }

This function runs automatically on the client and the server. First it runs on the client optimistically. The UI updates immediately. In the background it runs again on the server, authoritatively. Results are synced to the client in the background and the client snaps to the server state.

You can also extend this easily to do fancier stuff like automatically reserve a similar room at the same time.

It's very difficult to do things like this in a CRDT.

I am still excited about the work that e.g., Automerge is doing and follow it closely. But if you are trying to add realtime collaboration and/or offline operation to a classic SaaS style application, I think a tool built for client/server will often be a better experience.

See https://doc.replicache.dev/how-it-works#conflict-resolution

CRDT is not the only game in town, the alternative OT has been used and still being use for more than 30 years now by the industry[1].

I strongly believe that local-first software is the future, the sooner it becomes widespread the better because then we are not at the mercy of software company monopoly.

[1]Real Differences between OT and CRDT in Correctness and Complexity for Consistency Maintenance in Co-Editors:

https://dl.acm.org/doi/10.1145/3392825

This is the article that convinced me to build LegendKeeper on top of CRDTs/yjs. :) It comes with a unique set of challenges, but it’s helped me make a more user friendly, performant product. Plus local-first tech like CRDTs tend to get you collaborative multiplayer out of the box, too.

I had no idea Legend Keeper used CRDTs. That's awesome! What were some of the challenges?

To start with the good, building features in your app becomes super easy because your network layer is completely agnostic. Rather than making API endpoints for every new feature, you're implementing everything in the language of shared types like arrays and maps. The changes to those structures are simply merged with the server/peers and you don't worry about manually syncing stuff anymore. Everything you build now works offline by default, and collaboratively if you so choose.

Collaboration is where things get a little spicy. You're now dealing with multi-master replication. These challenges are exacerbated with multi-user use cases (though even a single-user experience is still multiplayer: one person using multiple tabs or devices is essentially multiple users in the eyes of most CRDT-based app architectures. You could also consider the server another peer.)

IMO, biggest challenge areas are data migrations, version management, data validation, and authorization. Sure you can migrate things on the server (if you're not building a P2P app), but you have to remember the server is now just a more authoritative peer; you can't guarantee your user will ever even talk to the server again.

If you don't have defensive data validation and stellar error handling, one peer can poison the others with bad data. It doesn't need to be a bad actor: you could simply add a bug to your code without realizing it, push it to prod, and now you've got viral data corruption. On the bright side, because your business logic is all in the client, unit and integration tests become super powerful. There's no need to mock a server and risk deviation from reality: the client is authoritative, therefore its tests are authoritative.

In my eyes, most of the challenges come from multiplayer. If you scope down your local-first app to a single-user experience + a server for backup purposes, I think things are pretty straightfoward. You'll still have to deal with version management and data migrations, but traditional desktop and mobile apps already have a rich history of best practices for this which can be emulated.

For data migrations / model versioning, have you seen Project Cambria [0], posted in 2020 [1]? (Also an inkandswitch.com project.)

[0]: https://www.inkandswitch.com/cambria.html

[1]: https://news.ycombinator.com/item?id=24699615

Do you use DexieDB for that app? I think I saw this app there. Cool stuff.

Yes! Dexie has been a great help with interacting with indexeddb

>>>

Seven ideals for local-first software:

1. No spinners: your work at your fingertips

2. Your work is not trapped on one device

3. The network is optional

4. Seamless collaboration with your colleagues

5. The Long Now

6. Security and privacy by default

7. You retain ultimate ownership and control

>>>

I love the idea of local-first software based on the name but Im having a hard time understanding these ideals. Skimming through the rest of the text doesn't seem to be helping.

What exactly makes software local-first? Like in concrete practical terms?

Simply put: your state exists on each client and the mechanism to reconcile that state across clients is not mission critical. Almost all apps currently are built to get their state from a centralized, remote location. This is easy to build but is fragile and not forgiving for loss of network connection.

CRDTs and local-first ideals means putting the client in charge of its state which leads to all these positive side effects: virtually instant UX interactions, privacy (your not sending data necessarily to a central server, and even if you are it could just be opaque, encrypted blobs that are proxied), network-agnostic syncing (email, bluetooth, internet, wifi, etc) and no fears of a service completely going out of business and losing your data.

Thats super helpful thank you!

Still curious if theres a side of this idea thats about local first software in a more regional or niche demographic sense. Like instagram but just for Oakland, or GrubHub but just for Austin. Im sure there are infinite problems associated with that sort of idea but still curious if theres some movement / push for that as well as this state-related stuff.

I think this is confusion based on “local”.

There’s geographical locality e.g. apps like ChatRadar[0] that show information from nearby.

And then there’s data-locality e.g. is the data stored on your device or does it all ultimately live on a server somewhere that’s the authoritative source of truth.

[0] https://apps.apple.com/gb/app/chatradar-local-discussion/id1...

Exactly. Your apps should work basically as well with no internet access, instead of either failing completely or becoming slow and sluggish.

I think that's the future of making web apps working fast. Trying to build some things now with Replicache (https://replicache.dev) and Logux (https://logux.io).

Replicache is a paid service though so trying to make it work with Logux ideally together with a SolidJS app I am building.

Thanks for pointer! Aware of Replicache, after quick skim Logus seems very interesting.

Thanks for trying Replicache!

Keep in mind that Replicache is free for non-commercial use and small commercial projects that are pre-revenue/funding. You only pay once you're making or have raised significant (>200k/yr revenue or >500k funding) money.

Building Replicache this way allows the company to be sustainable, and enables us to focus 100% on this product rather than having to do something else at the same time to make money.

I was fully unaware of this article when I posted [1] about my ideas on a Personal File System, and now I realize there is a lot of common ground between them.

I particularly like that this article gives specific, actionable advice to any application developer on how to improve their "local-firstness" (section "How you can help" towards the bottom).

[1] https://news.ycombinator.com/item?id=31527886

I think the combination of these "ideals" makes a lot of sense. If you want the combination of a web app but with native-like performance, you kind of get to many items on the list automatically.

The web app we're building now is kind of like an IDE (but for tasks/planning, https://thymer.com), and obviously waiting for round-trips in an editor is no good. We thought we'd just do "some optimistic rendering" but in the end found out that the combination of many of these "ideals" is actually really powerful:

If every millisecond counts when rendering, you want some sort of optimistic rendering and you can't wait for round-trips. Because of that it makes sense to have all data available locally, and have the client sync back changes when there's time. CRDTs are great for this, and then multiplayer support is kind of easy too. And now that you have data locally available anyway, you might as well cache it in IndexedDB, so it works offline too. And because now you want features to work offline (e.g. search), you need to do them on the client anyway, which means you can also have end-to-end-encryption.

With techniques like CRDTs becoming more popular, and web APIs becoming more stable (like for local storage, offline workers and built-in crypto), we'll probably see a lot more true "local-first" web apps. It's obviously not all perfect on the web yet though. Support for PWAs is still really limited, IndexedDB is quite a mess (slow, clunky API, and and too many different policies with regards to what gets deleted when by the browser), but it's getting much easier.

Spinners. The problem with many applications that store state across the network is that they do not have a spinner. Far too often I tap or click something and there is no indication whatsoever that something is happening, that my action was registered. Sometimes with modern styling I don't even know if the thing I tapped was active. I always think then that the developer did not perceive it would be possible to not have an immediate connection to the mother-ship. I don't even live in a place with bad mobile or home Internet. I have 4G network all the time, 1Gb fiber at home and the same thing happens over and over.

Spinners even in local context are often necessary. We just want to see them less, but I think that for an interactive application there must be an indication that the app is responding to what I do. In local context I could have a slow disk or even a network connected storage. Also spinners that spin for 30 seconds giving no indication of why am I waiting or what for make me sad.

Local first software is super important. I think there's a pretty big set of use cases for a non-CRDT model.

Instead of a CRDT, you can do the extremely pragmatic thing and take the lowest common denominator of various file sync systems(SyncThing, Android SAF providers, WebDAV, etc) as your example.

At any time you can:

* Read a file * Create a file * Delete a file * Rename a file * List files in a folder

And usually:

* Any file you make will be synced * You don't know what gets synced first * You don't know when sync will happen * A conflict could create a copy of the old file, or just delete it * The user may or may not fully trust all of the storage providers.

For any single-user app, or app where collaboration doesn't need to be real time, you can usually just use SyncThing or Dropbox or rsync.

What we need, then, is the equivalent for multi-user collaborative apps, some kind of abstraction for a "document" on a server, that people can edit, with permissions and all that, and that has document-scoped pubsub.

Perhaps some kind of sqlite over the web protocol.

Or, perhaps something like SyncThing optimized for multiple users and documents with different permissions.

Local first might be more popular if people didn't constantly have to reinvent it for every app, and there was one reusable abstraction people could use.

CRDTs are great, but not everything takes advantage of them anyway. Unless you can do P2P and lan sync, you often still have a single point of failure server.

I'm curious, it seems to me like the stated objections to CouchDB's approach aren't that bad, at least for quite a few use cases:

- Client optimistically goes to POST a JSON document. If new, there is no _id field included. - If there is no conflict, it's fine. - If there is a conflict, the server tells the client this. The client can try again, including the _id field of the current document version.

Could the CRDT process not be applied on the client looking to make an update? Or at least on the server side, where the client says, "OK, I've included the current _id, please merge my version"?

I have not attempted personally to use this at scale, but had prototyped a tool for small-scale collaboration on a project that seemed to function "well enough." No CRDT merging, but then there was no expectation of typically trying to work on the same sentence simultaneously. This was basically as a client/project assessment tool; any given project would start from taking a copy of a template DB on the server, then syncing and updating from that. From there, any sort of local-first changes were trivial, and easily synced back to the central location when back online.

> “Old-fashioned” apps continue to work forever, as long as you have a copy of the data and some way of running the software. Even if the software author goes bust, you can continue running the last released version of the software.

This is assuming they're not trying to phone home to a server and make sure your license is up to date.

I keep on thinking that someday I'm going to want to take a multi-month sabbatical from the internet; being able to keep on using Illustrator to draw stuff during this time is going to require finding a crack for it, as well as for the suite of plugins I depend on. Possibly a polite inquiry to the plugin developers will reveal the existence of a secret long-term offline mode available to people who've been helpful parts of the beta program for a while, or spur the creation of one, but I sure do not expect Adobe to cut me any slack despite being a paid user since the beginning of their subscription mode, nor do I expect them to release an official crack as one of their last acts before closing their doors forever.

I'm glad someone has written this down, a lot of us probably think this but don't bother.

"Local-First software." Isn't that just a desktop application that syncs?

The in-page anchors are broken in this article. Clicking on the linked phrases "later section" or "section on collaboration" don't move to those sections on the page

Is your page big enough to already display some or part of those anchors? happens to me sometimes.

It's small enough where it should relocate, but it doesn't

That's why though, you see what Im sayin? If any part of that div is in screen, not all browsers will refocus on an anchor, especially when there isnt sufficient whitespace beneath it to extend the tail of the page.

Reply on news.ycombinator.com