Local-first software: You own your data, in spite of the cloud (2019)

Love this and it aligns well with the work I'm currently doing.

I believe local-first peer-to-peer networks have gotten close-enough to being able to deliver feature parity with centralized Application Service Providers for 5+ years now, the industry just hasn't caught up.

For social media platforms like Facebook and X, their "killer app" features are all solvable with hash-based data structures (chains, trees, forests), p2p capability systems, and gossip protocols. Identity systems are solvable in ways that non-technical people can understand.

Part of the solution was the p2p stack maturing. The other part is that centralized solutions have normalized a lot of user flows that p2p can be competitive with, users have been trained up on a lot of patterns that previously wouldn't have been palatable in the market.

There is extreme market demand for it, and the zeitgeist is tuned into the societal and financial failings of ASPs right now.

Facebook is trying to act as the free relay and archivist for the entire world's social graph, there are a lot of hard problems to solve there which personally, I think contributed to social networking silently dying and degenerating into the monstrosity that Social Media is.

Once this stuff starts making it into end-user products, I suspect it's going to be a one way door. P2P can provide user experiences that centralized services can't match - either for technical, financial, or legal reasons.

We are in a sweet spot of opportunity for building the web as it was promised to us.

If folks working in this space want to compare notes, or are looking for work on a well funded no-BS R&D team developing this space, my email is in my bio.

My side project is a personal budgeting web app [0] based on these principles. The technologies to support this type of architecture are available and fairly mature.

For example in my budgeting app:

  - All of the user's budget data is stored client side in IndexedDB
  - Offline support via a service worker
  - Peer to peer synchronization between devices via WebRTC data channel
  - Lightweight IdP server which also handles the WebRTC signalling (via web sockets)

This setup is nice because the user's data never touches the server and very little server side infrastructure is needed (none at all if you don't need data synchronization). One downside with the data sync is that both devices must be online with the app open at the same time to sync, but perhaps this could be improved with web workers?

[0] https://dollero.app

As someone who very closely watches this CRDT and local-first space, it's really quite a bit harder than doing it the old fashioned client-server-model way, because you have to architect your systems to sync seamlessly when offline and then online. I've used tRPC and GraphQL before and I've really been thinking of something where we'd have the same DX as the client-server model such as with tRPC and GraphQL but in a way where there is no difference between client and server state, there is only "state." This makes sense because for local-first, the entire state of the app must be replicated locally anyway.

Recently there have been CRDT solutions that try to solve this problem, however, such as Triplit [0], or ElectricSQL [1].

[0] https://triplit.dev

[1] https://news.ycombinator.com/item?id=37584049

This article has been quite the inspiration for many projects and progress on this front. I think we're seeing more and more developments around CRDTs and local-first frameworks / applications.

I'm working on a few projects in this area:

- https://www.typecell.org - Notion meets Notebook-style live programming for TypeScript / React

- https://www.blocknotejs.org - a rich text editor built on TipTap / Prosemirror that supports Yjs for local-first collaboration

- https://syncedstore.org - a wrapper around Yjs for easier development

In my experience so far, some things get more complicated when building a local-first application, and some things get a lot easier. What gets easier is that once you've modeled and implemented the data-layer (which does require you to rethink / unlearn a few principles), you don't need to worry about data-fetching, errors etc. as much as in a regular "API-based" app.

Another interesting video I recommend on this topic is about Linear's "Sync Engine" which employs some of the local-first techniques as well: https://www.youtube.com/watch?v=Wo2m3jaJixU

Only three past submissions with discussions (though quite a few comments) the most recent discussion about two years ago:

https://news.ycombinator.com/item?id=26266881 - Feb 25, 2021 (90 comments)

https://news.ycombinator.com/item?id=21581444 - Nov 20, 2019 (241 comments)

https://news.ycombinator.com/item?id=19804478 - May 3, 2019 (191 comments)

I've switched to building VS Code extensions for my local-first personal side projects. The reason: no server to run and all my data is in plain text so I can put it in source control.

I've built and use a time tracker, a double entry accounting system (using a Beancount(ish) syntax) and an activity tracker. Two of the three of those were using older CLI code I wrote a few years ago adapted into an extension.

Adding a webview that tracks the content of a tab is pretty simple. It also gives me the excuse to write parsers (some of my favorite code) which allows me to render the text into a data structure that my webview code (in my case React) can render on each keystroke. I don't get fancy, I parse the full buffer on each keystroke and with proper React state management the DOM updates are optimized.

I don't think I'll ever open source or publish the extensions themselves as they are too specialized for what I need and want but I may (if I have time) generalize the webview + tab change/rename/delete/close tracking code into an open source library.

I'd recommend this approach if it fits with your side projects. Its an easy way to get a web gui for your ideas without adding to the graveyard of broken domain dreams.

I love this article and its ideas, but let's talk about where it falls short in reality.

1. CRDTs are a pain, especially rich text. Schema migrations are a pain. And eventually when you have a very successful product, you'll have issues with vector clocks that grow linearly and unbounded with every user who pops by to collaborate. If you're successful enough to have a 2000 person workspace, you won't be able to / want to sync an entire workspace of data in realtime to everyone's local machine, so the whole system devolves into a glorified local cache (nothing wrong with that, but it's not a CRDT and arguably isn't local-first).

2. Freemium SaaS has proven to be the most lucrative business strategy. Users get stuff for free which widens the top of the funnel. Students go into the industry and bring their tools with them, and then you charge enterprises for collaborative workspaces per-seat (I call this the Slack playbook). Given the challenges of CRDTs outlined above, you're making this job really hard.

3. If you're making all of your money on enterprises, they would prefer that a rogue employee can't save all of the company information offline when they leave. That's why you can't export your company Gmail or Notion workspace. Employees will pretty much always be online so local-first isn't really necessary for them, although everyone does want the performance of local-first.

4. There are large swathes of problems that are not suited for local-first -- basically any application that requires transactional writes. You can't enforce simple invariants like "every use must have a unique username". Almost every application has requirements like this unless you specifically design around them. Sometimes, they can't be designed around at all.

At the end of the day, I like to think of choosing local-first through the lens of the "technology budget". If you want to build something local-first, then that's your entire technology budget and then some. Do you want to bet your entire business on that? Or do you want to use tried-and-true tools and spend that technology budget on something else that moves the needle?

Developers love to know how something is built, but users really don't care so long as it works great! Example: Obsidian.md is awesome, and local-first. But it's mostly developers who use it -- everyone else just uses Notion.

I still love Local-First, but as far as I'm concerned, it's a developer honeytrap.

Is there any equivalent for apps where the code is open source and the data is totally under your control but in the cloud? I'd love something that's as easy to install as an Android app, but ran on a server someone else maintains. This is certainly possible now with enough effort, but there doesn't seem to be anything nearly as standard and easy as PHP on a shared host used to be.

Omnifocus does quite well with this. They offer a sync service, but it’s just webDAV syncing a few XML files so you can easily host your own if you want, or throw the files in iCloud.

People who are serious about this topic should start by studying past examples. Once upon a time all software was local first, and there's plenty to study there about what changed and why.

The masters of this game are Apple. Bar none. All their software is local first and asynchronously syncs to iCloud (optionally!) using CloudKit, which is a relatively consistent replicated database. This design is hard to master but worth starting with. They also support P2P that works (Bonjour and friends). Apple can do this because they're firmly rooted in the software development milieu of the 80s and 90s. They never got the memo about the web taking over and revel in writing high quality desktop apps. Users love them too. Developers like to pretend this ecosystem doesn't really exist outside of mobile, but it very much does.

There are a few things that make this hard for other kinds of developers to do well.

One is that it requires apps to have file formats, and most devs don't learn these days how to design them. They know SQL but not how to wrangle complicated byte arrays. And our infrastructure kinda sucks at working with files. You can't POST a directory over HTTP, for example. SQLite helps a lot here, but you still have to publicly document your schemas if you want the user to truly have a chance at "owning" their data. The cloud then just holds a backup of these files.

The desire for strong collaboration and sharing features is another problem. Apple isn't particularly big on those. However, a lot of software barely needs it! In most cases a way to publish or share documents with a hyperlink is good enough, especially if others can comment on it. A lot of users don't actually want to allow unrestricted real time edits to their work as it undermines ownership, hence why Google Docs has now moved to a more Word-like workflow where edits are help in suspension until explicitly accepted.

The biggest problem is that the only form of sharing that actually works well is an HTTP link, due mostly to OS makers catastrophically dropping the ball on anything else. And that pushes people towards the web. Apple solve this with light web renderers for their file formats that let people view read only copies, more or less. Often this is sufficient.

The Apple approach + an openly installable backend is a reasonable way to do local-first software. It doesn't imply the software has to be free or open source. It can still be proprietary, you just need to sell the backend as software as well as the frontend.

I'm building a flashcards app using Pouch / Couch DB at the moment. It's not done yet, but I wonder if making the switch to Automerge would be worth it?

Are there any resources to learn more about the differences it makes for a developer, when implementing a traditional Server-client architecture as opposed to a P2P approach? So far, I have only worked with server-client architectures, it's hard for me to estimate the pros / cons.

Also, is there an option for Automerge to sync via a central server instead? Any examples?

So has there been much progress? Is there, for example, a rich text CRDT editor that adheres to these principles?

I made a local-first platform designed for privacy, ease of use, and no vendor lock-in to sync and backup precious data using SQLite and CRDT.

https://github.com/evoluhq/evolu

“Sooner or later, everything old is new again.”

How do CRDTs work with blobs / files?