Local-First Software:You Own Your Data, in Spite of the Cloud (2019) [pdf]
martin.kleppmann.com
martin.kleppmann.com
E.g. you put a pin on a map to mark the parking spot, and want to share it with somebody standing right next to you. But it does not work because the connection to some data center 1000 km away is flaky. This is just plain bad design.
Or you want to print something or get two machines to interact. Any remote third party you bring into this interaction is going to increase latency, reduce bandwidth, increase the probability of failure, etc.
That is why we use the local first approach for factory shop floor software at actyx.com . You don't want your factory to stop because the link to the cloud is down.
It is also an issue in developed countries as soon as you leave the cities. And factories are often in rural areas.
E.g. in Germany, as soon as you leave the big cities, the options are atrocious. I live in a very rich and touristy village in the German alps.
I get 50 mbit/s DSL, which is actually just 25mbit/s down because the wires are so bad.
For comparison: in rural Romania, I get gigabit fiber to the premise for an eye-watering 10 EUR / month...
Exactly! Local copy is just treated as cache in those apps which offer offline support. They are called "Offline-also" apps, not Offline-first apps.
I miss those days when I could just buy the software and own it. Collaboration can happen in offline-first apps too, it's tricky but it can be done.
Everything is online-first web apps these days because of the reasons listed in section 3.2.1. There is one more reason for popularity of online-only web apps:
When startups are testing their idea, it's far easier to put together an online-first app than offline-first. It's easier to get traction comparatively because it just requires a browser. When they get some traction, then a desktop app is a low priority to them.
I want to bring back offline-first desktop apps back. I've already started with Brisqi[1] an offline-first personal Kanban app. Hope more people take an offline-first approach.
For similar reasons, I could write a whole essay on why we developers need to create a better open source cross-platform framework to develop offline-first desktop apps with permissive licensing.
The user has to install CouchDB on their desktop PC to use the local first option. Basically they just use a different html file to access the local CouchDB instead of the Cloud based CouchDB.
The app is also offline first and doesn't use the internet at all unless you turn on the "Live Sync" function in the app's preferences.
At that point it's still local first but it syncs with my Cloud based CouchDB and users can access that by using a different url. Since Live Sync works both ways whenever the Cloud CouchDB is updated those changes are pushed to the desktop PC if it's connected to the internet, or as soon as it is.
Users can also make backups and snapshots of their DBs on their desktop CouchDB.
I released ezInvoice back in 2002 and this is something I've wanted to do since then. Feels pretty good to have what's needed to get it working.
HTML version here
Other than a few highly-specialized niches (InfoSec, government, military), the de-facto reality is that online-first collaboration pros seem to outweigh the cons.
It's simply not true, or if it is, it's only because of what you're choosing to looking at to build your view of how it 'seems'.
https://mydata.org/ https://en.wikipedia.org/wiki/General_Data_Protection_Regula... https://ethereum.org/en/ https://solid.mit.edu/
Meeting every week: https://identity.foundation/working-groups/secure-data-stora...
> All things considered, data breaches are rare, and if they do happen, they're usually to be blamed on configuration blunders, not the services at large.
Did you work at Google? Do you know about PRISM? https://en.wikipedia.org/wiki/PRISM_(surveillance_program)#C...
Can't tell if you're intentionally or accidentally misleading.
All I'm doing is looking at the market cap of companies like Facebook and Dropbox. I mean, hell, Facebook literally sold personal data and sure, you occasionally see "boycott FB" movements, but they're all bark and no bite.
Only a handful really grasp the context of what is happening with their data and most importantly how their data could affect them.
People routinely do things against their own interest. It's up to the other people that care about this issue to make competing products that don't harm. There's simply no way to get everyone to really care about an issue with nuance and/or niche knowledge.
The mindset where we're the only ones that really understand what's going on and everyone else are just sheeple -- sheeple I tell you is gross. This issue really isn't all that nuanced. That's just the lie we tell ourselves because we don't want to admit that we're the nutters being like "they're watching you scroll through Facebook and they're gonna use that to lower your credit score. THE ALGORITHMS!!"
I'm proud to be part of this weird club but let's not use it as an oppertunity to shit on our outgroup.
Wanna keep in touch with friends? Gotta use facebook. Wanna watch cool videos? Gotta use Youtube. etc.
But there's a very strong movement of non-tech persons for privacy, decentralization and free software/culture. In France, it's best exemplified by the Framasoft association which started from a network of teachers : https://framasoft.org/en/
I know a number of people who dispair, thinking they can't protect their privacy. But gradually, those who know a geek may be moving towards better software, as privacy respecting options with decent UX get developed, and as their FOSS-geek friend or family member patiently points out tools they can actually use.
You eat your ham sandwich. But did you hear the pig cry out at the slaughter? The same people reading your comment right now are the same people sticking the knife in you.
"It is difficult to get a man to understand something, when his salary depends on his not understanding it." --Upton Sinclair
In 1998 people turned their back on the FSF and joined the "open source" initiative. Everything that has happened since has been inevitable.
In addition, there's a difference between using those services and being "totally okay" with it. If you like Evernote, then sure, you'll be using the cloud. If you had a way to use Evernote without storing your data in someone else's cloud, a lot of people would be happy.
Even non-technical people talk all the time about not trusting services because of data issues. Many will not use a new service because they're worried about their data if the service stops. It's a really big issue for businesses.
This is good news, as far as system architecture is concerned, as the opinion that matters is principally concerned with utility, convenience, usability, etc.
Secondly, the ship may have sailed, but interestingly these ships apparently are generational. There used to be a ship called AOL. /g
(There is a reason every ideological effort for authoritarian control heads straight for the school house and youth camps. The reason is the said 'ship' that sails, on schedule, every ~20 years.)
At one point the younger generation does not even remember there was a local-first software.
Sadly the relentless efforts of the corporations are paying off.
That doesn't mean people are okay with it. It means their choice is made for them.
There's nothing they can do, they have no realistic expertise or control, and they vote with what - dollars once every phone purchase?
And even under the model where printing money causes inflation, we're in a negative interest rate regime meaning printing money would cause /deflation/.
More and more, I am seeing many companies that realize they can do both: take the user's money and feed them ads. After all, that's more profitable than just taking their money.
As examples, T-mo, Verizon, my local gas station, various airlines all both take my money and give me ads. Some are even known to sell my data.
So, can I actually vote with my wallet?
Probably we will see the same play out here.
If you mix the two crowds: the people that actually care with the people that will just want faster apps, you get a powerful bootstrap traction.
Lets also not forget that things might just vanish and when we are able to collect them and make personal relations that make sense to us, we will want to be secure that things that matter to us wont go away because we are keeping them. Its pure psychology.
The parent poster is forgetting that before we get use of how things are right now in the digital realm, we had to deal with the material world and thats how we shaped our minds. And in that world we learn how important is to keep things nearby and have mnemonics to help our minds that needs to keep so much information that our devices become our second brains.
I'm working on a fancy chromium inspired prototype to tackle this exact issue and just with what i got here working in "prototype mode" i kind of don't want anything else.. it's just.. another level.
Its technologically superior and even ordinary, unaware people will just want this. Cloud have its place of course, but "local-first" will eat a lot of their lunch in the coming years.
Do we really though? Most people just buy a new phone, idly agree to all cloud sync stuff that the Android vendor has slapped on it and only notice "hey, I have 500GB free storage for the next 6 months!" and that's the end of it.
There is a market for local-first. Absolutely. But I question how big it is.
Maybe in corporate? If not there, I don't think you can make it big in that area. We at HN are a fairly negligible rounding error compared to most of the internet-connected humanity.
Local-first is a technological sophistication from the current tech status-quo, so it requires more labor to get at a point where it can stand against the current cloud-first applications.
We will only be able to see how far it can go, once the next-gen applications reach the same level.
It will be a harsh competition, but if we can just ship the platform along with it, and at least Android make it possible, users wont even notice, because it was the dev's that made that decision for them. And the dev's will do it because it might make their apps a little better, more responsive and fail proof.
Of course this all must happen before 5G or part of the leverage is gone unfortunately..
But we need at least try anyway, because a world where all the information is centralized on the FAANG titans is a world where freedom, civil and human rights are constantly endangered(especially in the age of AI automation).
And yes, the world is heading into a dystopia. All of us that care better prepare some big ZFS clusters for redundant data and well-connected ISPs so we can still exchange data in an encrypted and decentralized manner.
But I feel that it's a lost battle. We missed the train. There's still time but it is shrinking every year.
Increasingly, millennials also are having children, and I think there is a market opportunity for these sorts of products (e.g. I want to sort pictures by my kids, but without giving facial recognition to fb/google).
However I would've said the same thing about Telegram/Signal/Defi and other decentralized/privacy first technology not too long ago.
I think there's a reasonable case for the idea that as the conversation around censorship and data ownership evolves, we'll see a restructuring of the market toward some level of personal data ownership. We're probably far from that today though.
Exactly
Data breaches are not the only consideration. There's not many things more important to a business than their data and putting that entirely in a 3rd party's hands at a remote location only is very risky and could be catastrophic.
It's one thing to send a copy of a blueprint of a product design to China to have it manufactured and quite another to send them your only copy of it.
If both of those things existed you would be able to build apps like LinkedIn such that LinkedIn could just access your PDFS so your profile is actually hosted directly on your PDFS - so if you delete your profile info on your PDFS it's gone from LinkedIn. Of course, this also depends on trusting LinkedIn to not just copy over the contents of your PDFS but that could be handled as well.
What happens if you set up OneDrive and Dropbox (and Google and Apple) to sync the same folder.
Chaos, potentially. Think about deletes... delete a file from the web UI of one service, its client deletes the file locally, but that same instant the _other_ service sees the missing file and happily restores it, the first client deletes, the second client restores, etc. etc. until you've blown gigabytes and gigabytes of network bandwidth.
>Solid (Social Linked Data) is a web decentralization project led by Tim Berners-Lee, the inventor of the World Wide Web, developed collaboratively at the Massachusetts Institute of Technology (MIT). The project "aims to radically change the way Web applications work today, resulting in true data ownership as well as improved privacy" by developing a platform for linked-data applications that are completely decentralized and fully under users' control rather than controlled by other entities. The ultimate goal of Solid is to allow users to have full control of their own data, including access control and storage location. To that end, Tim Berners-Lee formed a company called Inrupt to help build a commercial ecosystem to fuel Solid.
https://solidproject.org/users/get-a-pod
https://solidproject.org/developers/tutorials/getting-starte...
If we're talking about the semantics of building desktop applications, then i think you're looking for freedesktop.org, although it quite POSIX-centered.
Higher-level, federated alternatives include ActivityPub (HTTP signatures + JSON ActivityStreams), XMPP (XML stanzas), Solid (HTTP + Linked Data).
All four of these standards could receive extension proposals for more semantics for specific use-cases. However, Microsoft (who owns linked in) and other evil tech-multinationals will never adopt a standard because that would allow competitors to walk on their turf. Remember when gmail.com chat was federated with Jabber/XMPP until Google pulled the plug?
Within the contents of the paper, it talks about CRDTs to technically implement it.
I'm not familiar with the technical tradeoffs but I bookmarked some past HN threads mentioning CRDT did not fare as well as OT (Operational Transform):
This has only really started being true in the last year or so.
https://github.com/automerge/automerge-rs
I'm hoping this matures a bit more in the next months, seems really promising.
To be clear, I think this is exciting stuff and your article is great. I'm just curious if there are any open research problems in this area that would make them more appealing in production.
So on the server I'd rather have a native implementation - which would also remove the need to bundle a JS VM in databases. And if we have that, it should be pretty easy to pack the same code into wasm for the browser.
And this is all assuming a JS-native web world. I also want CRDTs for native apps like Bear. And it'd be much easier to use a rust library than link to v8.
The following article shows that:
> In the Trellis project we experimented with a “time travel” interface, allowing a user to move back in time to see earlier states of a merged document, and automatically highlighting recently changed elements as changes are received from other users. The ability to traverse a potentially complex merged document history in a linear fashion helps to provide context and could become a universal tool for understanding collaboration.
Now imagine teaching all of that to a less technology minded audience. It probably won't go well.
[1]: https://github.com/pouchdb-community/relational-pouch#many-t...
I get that they want to inspire, to show data backing up the idea that people are open to that software model, and in general to give hope and inform but... I am getting way too cynical and I can't help but wonder if actually writing a tool that builds the foundations of such a movement wouldn't have been a time better spent?
I mean, at what point will somebody do something about it? And does it have to be your average overworked programmer who is sacrificing their scarce free time? When will one of these organizations that periodically toot their own horn about much they care, will hire several hardcore programmers to do something?
One of the authors wrote a book on this topic that gets high praise and frequent recommendations on this site. Several (all?) of the coauthors worked on prototypes on this topic that they discussed freely in order to enable others to follow their same path and explore it further. So yes, I'd say you're getting way too cynical.
Didn't know about the prototypes. Good on the authors! I do wonder why not one has been seen to completion though, do you know?
It's personal budgeting software that's local-first.
https://github.com/earthstar-project/earthstar
It's like localStorage but it syncs and can be used for collaborative apps.
The point is to avoid the economic incentives of commercially supported software, though, so indeed it's made by volunteers in their spare time.
There are use-cases for data ownership, stewardship, management and security.