Local-first software: You own your data, in spite of the cloud
inkandswitch.com
inkandswitch.com
- My data is stored at a well known URL on a machine that I own. If people don't want to host their own stuff, they can rent another person's computer.
- We need a standard protocol for "data that changes over time". This is a really obvious point once you start thinking about it - REST doesn't support realtime edits, and websockets / zmq / etc are all too low level. We need a standard way to express semantic changes (eg CRDT edits) and do catchup, that can work across multiple devices / applications / underlying protocols. I've been working on this as part of statecraft - https://github.com/josephg/statecraft but its still hidden behind all the documentation I haven't written yet.
- Then we need application-specific schemas to be published. Eg, there should be a standard calendar schema with events / reminders / whatever. Any calendar vendor could provide this. Then calendar apps could request on login from the user where the calendar data actually lives. Those apps could be web / mobile / desktop / whatever, because remember - we have a standard way to interoperate with data like this.
- Ideally the data would also be stored encrypted at rest. The server shouldn't need to be able to read any of the user's data.
You could build a peer to peer system whereby my desktop apps and phone share data with one another. But ideally, data should be accessible from any device at any time without worrying about whether your laptop is on or off. For that we need servers. You could make a single persistent server be a peer in a CRDT-based cluster of devices. That might be better - but its harder to implement and might run into issues with bandwidth and size (I don't want my phone to sync my whole photo library, etc). There are some generally unsolved problems here, but I don't think they're beyond us.
If you're working on this problem and want to chat, throw me an email - I'm me@josephg.com.
14 years ago I was syncing my contact lists from my personal server to a flip phone over vCard format (I think? Don’t remember exactly, but the contacts equivalent of CalDAV). Nowadays you either need to oAuth into Google or whatever other centralized provider a user might have, with whichever proprietary API they may or may not give you access to.
As with so many other things, a shiny new technical solution won’t change the reality that big companies are not incentivized to support interoperability.
With dat you have: - urls to individual files, with the understanding that they will change over time - built-in encryption and non-repudiability of every change - storage wherever you want, no peer is more important than any other
Shared object updates should be deliverable by any protocol that works for a specific application, whether client/server, peer-to-peer, or store-and-forward.
[0] https://www.inkandswitch.com/local-first.html#crdts [1] https://github.com/mafintosh/hypercore [2] https://github.com/automerge/hypermerge
(Except the encryption, I think.)
This already exists: it's called a DVCS.
We need something like a dvcs, but not exactly that.
It only really works when two of the sharing machines are online at the same time. I work around this by having a rasperry pi running 24/7 which does it's thing.
I always tried to avoid cloud based services, because I don't want to keep thinking about whether I can trust cloud providers and I kinda like the idea of a LAN beeing used for LAN things – you know.. local stuff.
I've set it up to backup folders from my phone to a LAN storage drive (running it as a service on Windows, with the storage connected as a network drive), but it seems to "forget" the connection between the server and the phone. It says the folders are up to date, and won't pull the most recent files and photos from the phone.
But the problem seems to be in the server; after restarting the computer or the phone, the connection between folders seems to be lost - it doesn't "see" the remote folder anymore, even if its ID has not changed.
To be fair, this doesn't happen with Syncthing only. Other tools like Nitroshare or Dukto are also hit or miss when trying to connect from the phone.
One hack to give this a try would be to play around with integrating APIs for cloud storage to go through a local cache system first, then do async synchronisation on demand. (Collaboration is not really enabled here but the rest of the principles in https://www.inkandswitch.com/local-first.html#practitioners become trivial). Essentially firebase but with developer facing API bridges, for dropbox, drive, Amazon, ftp, whatever. The twist: you have more control of the data and it doesn't necessarily go up to Firebase backend, the devices could even do the backend computations async. Then you can build on top of any cloud storage platform, even decentralize things. It gets even more exciting to think of decentralization.
I would like to additionally emphasize on the case for data being on the cloud where it is not necessary to be there. Location history comes to mind. I want it to be local-first, local-only.
Have found a couple really useful tools. My favorite so far is wallabag.
How to merge conflicts is probably the most important part in a non-trivial app. Does anyone know of examples or research that has been done in this direction?
This seems similar to what Evernote does
Opposite to the analysis of this article, I have chosen a webapp that runs in the browser.
The App itself is just a mhtml-file that includes javascript, css and images. See https://en.wikipedia.org/wiki/MHTML
Users can send it around as they want and do not have to install anything. Also the app makes a call to a server which makes recommendations on updates.
For the data-storage, I had some trouble because there was no database out there which supported replication, json-import/export and encryption. That was the reason I created one. https://github.com/pubkey/rxdb
The concept was blown off by my "Group Mentor" in Startup School last October, which was a bit disappointing, but it's good to see it being discussed here and maybe it will get some legs as time goes on.
I'll be releasing a simple app soon that, hopefully, demonstrates the advantages in a way that's easy to digest. At this point I'm not expecting much positive feedback though. It doesn't use any of the current trendy tech and there's nothing truly new or whizbanging about it. It is, however, fast, solid, easy to develop and modify, and runs on any server without having to install anything other than a web server.
My goal is to create a sort of log database merged with an app platform which maintains an encrypted and authenticated log structure that powers functions in developer code that update/manipulate local stateful resources. The apps on a user's device get access to resources managed by the platform like KV stores or Sqlite databases. The App uses these log messages, which are filtered and provided by the platform, to update these stores locally and creates new logs on the user facing clients in order to make things happen both locally and remotely once the logs are synced.
I am convinced that "logs" are the future for personal data.
Also note that Lotus Notes [1] has been doing all this replicated encrypted app platform stuff since the early 90s.
I am working on a log-centric bitemporal database at JUXT which I think intersects with this problem space as well, see my recent tweet relating to the article [2].
[0] https://www.scuttlebutt.nz/applications
[1] https://web.archive.org/web/20170228160130/www.kapor.com/blo...
Reading [1] makes it seem to me like I might just be reinventing Lotus Notes.
Only a few of us run servers, most people just run Scuttlebutt on their device so that they can their data as well as their friends' data:
- Windows, macOS, and Linux: https://github.com/ssbc/patchwork
- Android: https://gitlab.com/staltz/manyverse/
- iOS: WIP
As much as I would prefer local-first as well as offline-first approaches, I reckon that the future will only have a place on the fringes for them.
The reason is mostly the human cost of maintaining infrastructure, and the global lack of good people have the knowledge to do so.
Edit: Trusting hundreds of SaaS vendors vs. one major cloud platform is another debate though.
Saving costs is a major drive, I agree...
Saas software is not only an economic incentive, but also a UX win!
The user no longer need to update his software, you can deploy breaking changes (or security fixs) in seconds (as opposed to weeks, waiting that all users download the new release).
Security: the user no longer need to download random software from the internet, and a random malware on his computer will have hard time to access his online data.
I'm sure there is a lot more of advantages of the centralized model, but I feel it's unfortunate that they are never put under the light.
Obviously a new patch won't be far away, but there is a certain stability to controlling the upgrading process yourself.
> The user no longer need to update his software, you can deploy breaking changes (or security fixs) in seconds (as opposed to weeks, waiting that all users download the new release).
The user can also no longer choose when to update the software, and you can deploy breaking changes in seconds. Meanwhile, on my computer, I can choose which software to update and when I want to do it, and I'll do so as to not impact my work flow when I don't have the time to adapt to the latest and greatest.
> Security: the user no longer need to download random software from the internet, and a random malware on his computer will have hard time to access his online data.
On the other hand, malicious parties interested in many users' data now have less work ahead of them. Some shitty engineering at LinkedIn and suddenly millions of users have their data leaked.
It is true that centralizing the data means that a team of experts can manage the security as opposed to an amateur like me, but it's been proven again and again that it's unreasonable to expect data that you share with a centralized third party to be secure and private.
Everything is owned by the end user. This is only providing a recipe for people to use.
I will also mention that https://www.stackbit.com/ is doing basically the same thing but more from a “Make life easier for Website designers” perspective.
If you can send my data into the cloud you should also give me the ability to easily mock your cloud API so I can also send the data somewhere else...
The examples (MS Office, Trello, Dropbox, etc) also seemed strange to me: I'd think that neither an average MS Office user would care about privacy, data ownership, etc, nor an average nerdy user who cares about those would want to use something like MS Office or Trello. Then there's plenty of easier to solve and related issues that aren't yet solved (e.g., plain offline usage of some software, more widespread asynchronous collaboration), and the article talking about privacy and data ownership ends with "We welcome your thoughts, questions, or critique: @inkandswitch or [email protected]". Looks like a nice summary, but maybe a bit otherworldly.
I'd be down for working on projects in this space. I'm presently contributing some work into the Lean theorem prover where I'm hoping, with a bit of elbow grease, it will be fairly low-cost and attractive to build out more p2p protocols and libraries that meet our privacy and security demands.
It's an app I wrote primarily for myself, but it's been great to hear from lots of people who really "got" the vision themselves and use it a lot.
I simply love apps & sites that make immediacy and the feeling of "control" core values of the UI. Wish I had more time to give Mindscope more development love than I've been able to lately...working on that.
The reason that software is online is a business one, not a technical one.
Software as a Service is impossible to pirate and generates continuous income rather than a single upfront fee. That’s all you really need to know to understand why there is less and less desktop software coming to the market these days.
So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
True, but I host my data in the cloud because I don't want to pay the energy bill of the equipment at home. Despite not having concrete facts I believe it is more environmental friendly if we started hosting servers centralised in facilities instead of everyone doing at in their homes.
Then you pay someone else's energy bill.
> Despite not having concrete facts I believe it is more environmental friendly if we started hosting servers centralised in facilities instead of everyone doing at in their homes.
It's true that running 1000 instances of a software on the same big computer consumes less than running them on 1000 different computers, because they won't use all its resources at the same time. There's also less overhead if you can share their computations.
However, hosting everything at the same place has some environmental issues too. A datacenter needs cooling, and not all homes do. If there's a lot of data traffic between you and the software, that traffic needs to go through the internet instead of being local.
I'm not saying one way is better than the other; it probably depends on the kind of software we're talking about.
It dehumanizes the customer.
Anyway I don't think piracy is the problem per-se, but the question is: what is the commodity?
Because take Netlify for instance - picking on them, but you will see this sort of pattern everywhere:
1. Generous free tier to get you hooked. 2. $29 / month for something that costs $0.01 in terms of cloud compute. 3. $100 / month for something that costs $0.10 in terms of cloud compute.
Etc.
So really they are creating a fake scarcity and mark up stuff ridiculously.
But with local-first as I see it, you will get stuff like "filling in forms" or whatever software that has made companies into multi-million companies for free. And you will deploy that on cloud(s) of your choosing with no effort, and pay their pretty cheap per-transaction costs. Stuff that is trivial will rightfully become very cheap and commodity like.
This is probably a good thing for consumers, because for cloud companies and new startups to compete they'll need to offer truly innovative products. Do stuff that was really not possible before. Quantum computing for example.
If you really think the profit margin is 99.96% for that tier two customer, it should be pretty easy to create a competing business to provide the trivial service and charge a lot less. Like $0.02 and have a 50% profit margin :-) But I think you're underestimating how much it costs to run a business, even if compute costs are cheap.
The tech ideas talked about in the article could help commoditise at the saas level. You’d no longer need an engineering team because the app is designed to work with generic cloud services. The engineering is done by digitalocean or whoever. The software is free or maybe $100 one off license for forms say. Like a desktop application of yore.
It’s a separation of concerns between features and infrastructure. A bit like desktop apps. You don’t need to buy a new computer for each app, so why pay a computers cost in cash every year for form filling or value add file storage or whatever.
The major cost for paid tiers is not infrastructure, but user support.
Service as a Software Substitute (SaaSS)[1] is also inherently spyware, an increasingly popular feature as surveillance capitalism[2] infects our economy.
[1] https://www.gnu.org/philosophy/who-does-that-server-really-s...
[2] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...
As an example you offer an online document cleaner, the user uploads his document and you do the magic on it, but if you encrypt the document then the backend can't read it. The document needs to be decrypted somewhere.
It’s a genuine question for which I’ve done a small amount of google/Wikipedia reading, but I suspect the answer is “no, of course not”, but I’ve been surprised before by tech/crypto solutions that I didn’t know the details of.
I try to do my part. I do not buy access to games on Steam, movies and TV series on Netflix, books on Amazon, I do not use Google, I do not use Facebook nor am I in contact with anyone who does, and so on.
I am open to discussion. Why is it specifically capitalism's fault? What other economic systems would be favored, and why?
> If you do not like a service, you can opt out by not using it
This is a simplistic objection, that doesn't stand to cursory scrutiny. These systems invade everything, and give you no real way of consenting. When you visit some random site, you will be informed upon to dozens of services, with no way of even knowing which. You might not use Google, but can you avoid sending an email to anyone using Gmail - particularly those who use it with their own domain?
And this is getting impossible to escape even in real life, with companies employing many mechanisms to surveil you on the public street, without you even entering their private property. How do you opt out of that? Stay at home 24/7? No, that doesn't work either.
> If you have an issue with something used by millions of people, then you should consider the possibility that it actually may be the people who keep the service alive.
People are driven by incentives. We can criticize a system, government or company for incentivizing behavior we consider harmful.
> I try to do my part. I do not buy access to games on Steam, movies and TV series on Netflix, books on Amazon, I do not use Google, I do not use Facebook nor am I in contact with anyone who does, and so on.
And yet, the last site you submitted to HN loads stuff from Facebook's servers. Even if you have enough know-how to block them, you've helped them surveil other people.
> Why is it specifically capitalism's fault?
It's capitalism's fault insofar as it promotes these business models.
> What other economic systems would be favored, and why?
Criticizing a system doesn't mean one advocates its replacement.
These systems invade everything, and give you no real way of consenting
When you visit some random site, you will be informed upon to dozens of services, with no way of even knowing which
You might not use Google, but can you avoid sending an email to anyone using Gmail - particularly those who use it with their own domain?
with companies employing many mechanisms to surveil you on the public street
Consent requires intent. Just because my browser happened to connect to some website, does not mean I give consent to have my accesses recorded and used for other purposes.
You may say it's legitimate to log and track without consent, but you can't invent my consent.
And the fact that it's often considered legitimate to log and track without consent is what enables most surveillance capitalism (though some consensual business models exist as well).
> When you send a regular snail mail you don't control either the postal box used -- a company could be scanning the document.
And that would be part of surveillance capitalism as well.
> The issues lies in the fact that the streets are public and not private places
Maybe so; analyzing such questions is part of the studies on surveillance capitalism.
Sure, but it's how business in the real world works as well. You can't enter a shop without being observed or even recorded, either by cameras or the shop personal / owner in smaller ones. Your behavior during a conversation is analyzed in realtime and reacted to, that's the basics of selling. There is no invisible commerce in the real world.
That's a fundamental reality that you can't just omit, doing so would be irrational treating the web as something purely abstract, which it isn't. It's an abstraction / extension of the real world, because it's part of it.
What the web needs is obvious boundaries and private spaces that are basically extensions of your own home and those of your friends, make it clear that you can't do x or y (going in public) without being observed and analyzed. Right now the only real indicator is the tracker count of adblock plugins.
But there are plenty of situations where the analogy shows how invasive surveillance can be. Imagine if your car mechanic kept a log of everywhere you drove. Imagine if the mattress maker knew everything that took place in your bed (for product improvement, naturally).
I'm alright with necessary data collection, like the banks and brokers recording what I ask them to do. But that's because there's no other way of doing those things except for services or becoming a professional myself. If the software can work as not a service, then I'm less okay with the data collection when it's only offered as a service.
So the best thing would be to fight for better information, education and clarity about these things, instead of treating only the symptoms with regulations and laws that only excludes people from participating in the market.
Why not have mandatory indicators about different privacy levels and a driver license for tech that gives people the necessary foundations? This could solve the privacy problem from the ground up.
There are SaaS business models with local/native desktop software, e.g. Adobe CC.
iOS AppStore-monetized software can run locally and be licensed as a subscription. Apple encourages local computation and continues to add both hardware and software for on-device processing.
x86 clients are adding SGX (hardware secure enclaves) which provide stronger mechanisms for endpoint license enforcement. This can support multiple business models, including subscriptions.
Almost every OS vendor is investing in deployment and packaging, because it's perceived that users now expect either zero-install or app store experiences.
The issue is that most OS vendors are incentivized to make themselves gatekeepers and insert themselves into the delivery chain, for financial or security reasons. That always causes conflicting requirements. It's interesting to look at Flatpak, which is not built to privilege an OS vendor, versus almost anything else.
I don't like the privacy problems this generates, but in a business environment I want my stuff to work for my customers, and that means I do the hosting.
"No install" is a major selling point, unfortunately.
I'm not sure that's entirely true, but since most people are now familiar with the likes of mobile app stores or one-click/one-command installations that Just Work(TM), the insane complication and risk that come with trying to install and manage software on a major desktop platform like Windows is now obvious for all to see.
For B2B products, it's a huge win for driving market penetration on otherwise locked-down work computers. While development and IT sometimes get exceptions easily, the majority of business users have pretty locked down machines with minimal flexibility in installing software.
For consumer markets, you're right that it may not be as big of a factor, other than user expectations on the Just Works(TM) model from app stores.
But as far as using unapproved SaaS, they won't officially allow you to. But it does change the power dynamics at play considerably, since it's a matter of compliance rather than capability.
An unofficial piece of desktop software is a non-starter, since simply installing it would require notification to and the assistance of IT for the majority of corporate workers that don't have the proper privileges to install it.
An unofficial SaaS, however, can easily fly under the radar of regulatory and legal compliance teams, either via free tiers that don't require payment or via small enough T&E expenses that they never get noticed above the user's immediate reporting chain. This puts a soft limit on the amount of friction and red tape that IT and legal teams can put onto business users, because if you make it too hard for them to get the tools they need to do their job, they'll just go around you.
That leads to a lot of interesting dynamics.
- My current company uses Skype and Microsoft Teams for internal communication, but the way it's configured makes it super unreliable. IT has repeatedly expressed that the issues are user error, and not their problem. So we have entire offices and divisions that use unofficial Slack communities, some paid and expensed and some free. And it's already so embedded in various workflows (including client facing ones) that IT can't rote block it. It's now forcing our IT team to evaluate Slack as an official vendor, because we're using it anyway and they have no insight into usage (which leads to those regulatory and legal concerns you mentioned).
- My current company uses Exchange for email, but routes everything through Google for spam quarantine services. Our company email is a registered GSuite account, but every service (except the automated quarantine) is disabled because they don't want us using the other products. The division I work for does client consulting, and I need access to Google Ads, Google Analytics, and Google Docs (when a client dictates) to do my job. And many clients don't want to give access to personal GMail accounts. The solution? The account team that was most impacted is paying for and expensing an entire second, unofficial GSuite account on a random corporate subdomain they already had set up. IT even nudged me to that account team when I raised a stink about access needs. Turns out over 25% of the company is using that shadow GSuite account, and we're beholden to the good graces and P&L of the account team that's currently eating that cost.
- It's hard to police this. Even if you try to implement internal controls on the financial to catch these sorts of things, there are always ways around those. For example, I've seen plenty of clients leverage several of our existing agency retainers or contracts to have us acquire solutions on their behalf (which we use on their behalf, but also expand access to them) and just bill it as a passthrough cost against the existing spending authority. This would be a non-starter for desktop apps, but works really successfully for SaaS.
Well, yes, in the sense that someone might manage to access such a system despite any corporate security barriers if, as you mentioned, they initially managed to fly under the radar.
Of course, if caught doing that without the appropriate authority in the kind of environment I was talking about, that person would mostly likely be fired immediately at best. Potentially they could be sued for more money than they would ever earn if they wilfully caused compliance violations that resulted in big financial penalties. Potentially they could even face criminal charges that could see them imprisoned, depending on the nature of the violation and the jurisdiction in which it occurred.
Personally, I wouldn't consider it ethical to try to attract additional business in that way. As a practical matter, it's also highly unlikely to succeed with customers in sensitive sectors like finance, healthcare or defence/security work, though of course those are organisations towards one end of a spectrum and the IT departments in less security-sensitive organisations wouldn't necessarily be deploying the same level of countermeasures.
While your reasons make sense, they make sense from the supplier side. However, the ultimate decider of what makes "business sense" is not the supplier, it's the customer.
You can have all the supply-side advantages in the world, if your customers decide they want that other gal's product, you're toast.
Please sir, can I have some more?
With local SW, you can just write it and mostly forget about it, as long as OS stays compatible with your program.
Apparently they had bought several other smaller companies, integrated their software into their own program when useful, built huge servers for their cloud service that they also rent.
I knew the program was good - I was familiar with previous versions - so it was kind of a no brainer. The problem was the new cloud thing.
I am in favor of economic rational practices in agriculture (in addition to the normal functioning of markets), so that crops are chosen wisely between farmers and prices do not plunge because of all-or-nothing decisions, and sharing information between farmers is essential.
But data harvesting (no pun intended) of farming practices to the purpose of informing large companies and futures markets brokers of their possibilities can work against the interests of farmers and consumers. The way large corporations treat small farmers is ruthless and they negotiate no terms.
So if my and other farmers' data was being uploaded to the cloud without possibility of local storage, who had access to it? Every crop, production, pesticide, invoice, price was going there!
They wouldn't sell it, they told me. It is stored with e2ee, they assured me. If I ever forgot the password, data would be irrecoverable, they said.
But there was also the problem of internet connection issues, because farms are remote. Would it ever work?
The first thing I noticed after installing the program was that there no important message was shown that informed the user of what would happen if the password was lost. I typed in some new data, saved and exited. Then I started the program and reset the password with ease. There's no possibility of 3FA, either.
So I told them I could not use the program because of my poor internet connection, which was true. They installed a local version that was temporarily available.
But they took so long doing this that I developed my own modular managing system, based in - yes, that's right - spreadsheets. It fits my needs, it is completely portable and it works.
The formulas are getting cryptic, and there are still some things I have to do manually (like sorting lists or generating pdfs), but the files access each other and it works. I have no time to code anything and I don't think I have sufficient knowledge of database software (I am trying to find time to learn MySQL).
Maybe some day I'll make some GUI that handles all of this, in such a way that if everything else fails, I can still access and edit files manually.
https://a-gentle-introduction-to-sql.readthedocs.io/en/lates...
SQLite is a bunch easier to get up and running than MySQL, and the data is all in one file you can copy around. So, easy to backup. :D
If you do eventually hit it's limits, it's big brother is PostgreSQL, which is a really powerful database system. :)
So parent should probably be fine with a solution like this.
Portable, local and without the server configuration hassel.
At least you could 10 years ago, very easily (it was OOo.org I was using then I think) - I made a simple customer record system just 4 or 5 tables but the address "table" was a kind-of view of the spreadsheet for email-merge.
Presumably Excel and others do this too.
The SQLite libraries seem to be built into Python, so nothing extra to add:
https://docs.python.org/2/library/sqlite3.html
https://docs.python.org/3/library/sqlite3.html
And the PostgreSQL equivalent:
Hmmm, there seem to be several PostgreSQL Python libraries around, all in active development.
You'll probably be fine with SQLite though. :)
If you ever need a helping hand. Maybe with visualisation or analysis stuff shoot me a note.
Why does a problem have to be 'technical' to have a technical solution? Skype, Wikipedia, Amazon, eBay...pretty much every major technical product I can think of solves non-technical problems, almost by definition.
> But nobody is going to. Because it makes no business sense to do so.
You are talking about a general trend as if it's ubiquitous. Yes there are strong business reasons why most software vendors have shifted to SaaS, but that doesn't obviate the fact that some users have problems for which local software is a better fit. The local software market may be smaller, but it exists, and it's incorrect to say it makes "no business sense" for anyone to operate in it.
You only need to support it, no need to worry about clients/customers still using a 6 year old version because they refuse to upgrade for some reason.
My last company had at least one client lagging behind by about 100 releases.
I think it was browser compatibility and their internal enterprise firewall.
If you work in regulated environments this can be s big deal and this validation process becomes captured (forced) by the saas provider.
If you ran your own you validate and keep that unless new requirements. No need to revalidate every time a new release is out and pay for that re-validation.
I've got Blender open on my desktop. Could you explain the business sense behind Blender exisiting?
Makes perfect business sense.
I guess you are hung up on the fact that OP used hyperbole to say that "nobody" is going to focus on local-first software and you are trying to win an argument by taking this statement literally.
Can we be a little bit more charitable here, and actually argue the point of OP? Even though I am a die-hard Linux-on-desktop, Thunderbird-for-gmail, SaaS-averse gray-hairing dude who still is bent on spending countless nights and weekends setting up my own self-hosted services on my own bare-metal servers, I totally understand that the growing of SaaS-companies are due to economic reasons, not technical ones.
It is not like people are going around saying "Gee, I made this software here and I wish people could run on their computers, but they can't so I will have to create a startup business that runs this software on the internet and I will collect subscriptions as a way to fund the operation".
Well, I was mainly responding to the closing paragraph.
>So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
I could be more charitable and invent other things that I think they meant to say rather than focus on what they said. You are not the first to suggest it is a failing of mine that I do not do this.
Personally I think that if someone says say "But nobody is going to. Because it makes no business sense to do so.", it is reasonable to point out that there are plenty of motivations beyond 'business sense', and to point this out is fair comment, rather than getting hung up on anything.
If I am not the first one to point out your failure of understanding the Principle of Charity, do us all a favor and focus on that before continuing discussion with others online.
The point at which I start to think it might be being misused is when it is not being used to look for a more gentle version of what could be unintentional ambiguity, but rather is being used to change meaning from the literal sense of a sentence to something in favor of a current position in a discussion, as this invites people to try and have their cake and eat it.
If it makes it easier for you, forget the "nobody" paragraph or substitute for "a very tiny minority compared to the total population of people/companies that make software for a living".
With this in mind, think of how many companies would actually benefit from focusing on "data used by the software is local-first" vs "the software should get things done faster/help the user make more money/help the user get laid"?
If you actually argue that companies in fact would benefit from focusing on this but can not due to some technical limitation, then we would have a legitimate point of discussion. But it seems you can't, so all I am getting from you is this insipid argument over semantics.
It would be great if we could actually get to the point of OP after all this time. How about we try doing that?
(The same could work from the demand-side of the equation as well. If you argue there is a significant amount of the population of software users that say are more concerned about the software "to be local-data first" than "the software must help me do my job faster/make me more money/get laid tonight", then you have a point. And no, pointing out to Richard Stallman and saying "well, actually there is this guy and a bunch of other aspies that don't care about getting laid or more money" is not really a valid refutation of OP's point)
I did just do that in my previous post and noted that my point does not change, but hey, please go right ahead and keep telling me what point I should be discussing and what it is that others meant to say.
You might say "price" but then that's part of being better "better excluding financial considerations" is quite a different notion.
A limited list of reasons they may use it include available talent (people often learn on free stuff), availability of support and consultants locally or at their price, specfic features of one tool need or not needed, availability of developers to contribute/fix bugs, desire to have input handled openly and open governed. They prefer the UI or APIs. Flexibility to integrate more openly. Desire to make a statement in support of FOSS or specific project.
Reasons why they may not us FOSS, liability (someone to blame if there's an issue, spread insurance costs), privacy when dealing with issues, availability of consultants or training at the desired price, availability of talent (many people learn paid software at school), availability of managed server farms. They prefer the UI or APIs. Available APIs that have simple integrations. Etc.
Actually it's not true that local-first software cannot be businesses (or non-profit orgs but still revenue-generating), it's just that the money to be made in local-first is an order of magnitude (at least) smaller than with cloud software. On one hand, the money to be made is much smaller, but on the other hand, local-first is where the next big disruptions can happen, while still having a non-zero revenue. Look at open source or open data projects for some examples on this: Wikipedia, VideoLAN (VLC), OpenStreetMaps, etc.
This may be a scenario where it's possible to create a communal good (for personal data ownership and control) which doesn't require businesses to lead the way.
In addition, it is feasible to distribute some-or-all of an application's software via the cloud while still keeping application data locally; we can be more nuanced here and keep a lot of the business benefits while caring more for users.
But there is a case to do so: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
> we can be more nuanced here and keep a lot of the business benefits while caring more for users.
If you are talking about "business", then we should not talk about "users" but "customers". And it so happens that most "customers" (so far, at least) consider it more valuable if the business also is responsible in keeping (and securing) the data. For these customers, putting the responsibility of controlling the data on them is a burden, not a privilege.
I'm stretching a bit here, and haven't mused on your thoughts a ton, but perhaps one reponse is that this shift places demands on ensuring that local storage defaults - and the security of the delivered software - are adequate to keep the data safe in the hands of a layperson user/customer, and that that needs to distribute the cost and attacker opportunity away from massive centralized data silos enough to make the markets (and users, customers) follow.
Another angle would be to (optionally) assign another organization/entity as your storage provider - I think the 'Solid' project was suggesting something along these lines.
Sounds very cynical and short sighted to me.
Pros to customer: Nothing to install. Can be used on any operating system with a web browser. Updates and security patches are performed automatically. Can share documents and collaborate more easily.
Pros to developer: Easier to support multiple platforms. Easier to upgrade users and migrate data.
> So yeah, sure, if you were to build a piece of desktop software from a clean sheet of paper today, this is a really good guide on how to do that. But nobody is going to. Because it makes no business sense to do so.
Imagine having to install Trello, Google Docs, Slack etc. manually everywhere you wanted to use it, deal with updates yourself and ask people you wanted to collaborate with to do the same. That makes no sense in terms of ease of use.
I imagine it's similarly easy with chef/puppet, etc..
One of my problems on Ubu is that it's so easy to install stuff that I can acquire a lot of unused cruft; mind you it's mostly a mental problem now with disk sizes as they are.
Desktop software works "forever" (as long as the underlying OS and drivers don't break support for it, at least) and the longer you use it, the more value you get for your money. SaaS works until the company that maintains it stops supporting it, and the value scaling is basically the reverse of desktop software.
That's the way things worked for years, and the way most software still works (except through app stores) and it wasn't a madhouse of anarchy and suffering, it was perfectly fine.
Installing software "manually everywhere you want to use it" implies installing it once in 99% of cases, maybe twice in most of the rest, where "install" is a trivial process. Businesses will require that everyone use the same set of installed software, so that's not even an issue to begin with. It's not unreasonable to expect people to have a PDF reader, or Adobe software if you're working in graphics, for instance, or for Windows users to be able to read Windows documents.
Plus... you do know native software can handle automatic updates and security patches as well, right? And if you don't think people can share documents or collaborate easily with native software, I mean, that's been a thing since Napster and IRC. FTP and USENET. SAAS didn't solve any of those problems, because they were never problems.
It's not trivial at all compared to "click this link (and optionally sign up) to start using the software". Especially with the landscape now, asking a likely non-technical user to download, install (potentially with security warnings) and launch software is a lot to ask and will impact your conversions if you're not an established product.
With the landscape now, installing and launching software involves clicking on it in the app store and waiting a few seconds for the installer to finish.
With the previous landscape, installing and launching software involved, for Windows, downloading an installer, double clicking on it, maybe going through some default settings, and waiting for installer to finish. For Linux, instead of downloading an installer, you pull it with a package manager, maybe go through some default settings, and maybe wait for the installer to finish.
It was never that complicated unless you were compiling all of your software from source, but that's an extreme fringe minority in the whole universe of software consumers.
The process of "installation" in the SAAS model is merely replaced with registration. Instead of downloading the program, you're uploading your credentials. The complexity is still there, because it has to be, but it was never that difficult for the end user to begin with. And the process of registering for a site is, arguably, more complicated than native software in many cases. I don't have to deal with individual passwords for each of my native apps, for instance.
Even when it really is that simple, any extra steps are going to increase drop-offs.
> I don't have to deal with individual passwords for each of my native apps, for instance.
Single sign-on solves this and you can create web apps that don't require initial sign up to use. Any desktop software that allows collaboration is likely to require a sign-up process as well so the desktop solution is always going to be more complex to set up.
Now days, you have to figure out which of the 10 download buttons is the real one. And then you have check hashes if you want to validate that you actually downloaded the installer that the publisher released.
I think that browser based software is a huge step backward in terms of richness of user experience. I can't believe that, in the year 2019, GSuite has usurped Outlook and the traditional native office applications in the corporate environment that i work in. But here we are.
But, there's no denying that any friction for the user getting the software will drastically reduce their likelihood of using it at all.
I'm sorry, but the fact that the dominant paradigm for software use has been, and still is native (I'm counting apps as native here, because they're installed, even though in some cases apps are just wrappers around a website) disputes it.
If people want to use some software, moreso if they need to, they'll put up with installing it. What you're describing is really only an issue for web services, and would maybe only =apply to native software that worked according to a subscription model, which most doesn't.
You are assuming that the software in question is both packaged for the distribution you use and that it is available from the default servers for your package manager (and that the packaged version is recent-enough for your purposes).
The software installation landscape for Linux is a lot more complex than this, and for rather a lot of software, these assumptions break down.
Here are the Linux install pages for just a few examples of popular end-user applications that are actually available for Linux:
https://calibre-ebook.com/download_linux
https://www.dropbox.com/install-linux
https://www.openshot.org/download/
Snap and Flatpak promise to simplify the provision of binary installers for Linux so that developers don't have to create both a DEB and an RPM, much less an install script that has to be run as root, but that ends up pushing users even further away from the package manager, not closer.
If your only objective is making money, that's correct. If your aim is to achieve something more, you may still want to take this risk in spite of knowing you'll earn less.
And in Oracle's case, you're neglecting to mention their ability to bring lawyers to bear on any infringers, something not a lot of software companies may have the capital to do, compared to a SaaS model.
Buying a piece of software (one-off) almost always requires budgeting and capital-expenditure approval. Add in the mess of depreciation calculations and maintenance fees, and it's just a mess. Can be weeks of work just to buy £200 worth of software.
A subscription service usually gets budgeted and booked as recurring expenditure, and looks better on the balance sheet.
An OpEx increase requires an increase in budget paid for by taxes, while CapEx can be covered by a one-time bond.
This leads to paying vendors millions for a software+support contract rather than hiring a few engineers.
That's only partly true. Software gets more and more developed for the big corps in mind as the sole users. So web browsers are developed to make Google and FB happy. Kubernetes is developed to only work in AWS and Google Cloud really well. And so on.
All that hollows out the foundation that gives software usability to all of us. Eg. if big linux corps only focus on Kubernetes, Kubernetes only works in AWS and GC, then nobody puts effort into Linux anymore. It might even happen that linux gets unusable if you don't run it in such kind of environment. In the end it all depends how the majority of resources get spent.
So we come from today's tech level X everywhere to tech level X+Y in the cloud and tech level X-Z on local clusters. Then where will most of the data be? Where will the new hip stuff be? How can you develop new stuff with an incomplete local infrastructure?
What you lose is not the desktop software but the desktop. And you will lose it in a technical sense. You won't be able to rebuild it technically alone.
I'd argue that the increasing complexity of the stack involved in any given product has also brought us to a place where the kinds of thorough QA we had 20 years ago isn't really viable anymore, at least for most smaller orgs.
The ease with which you can fix a deployed product in a SaaS situation is a huge boon.
Apparently there are needs, that's why this guide even exists. Maybe not many but there are people willing to pay for better privacy/data protection.
But you are right, this won't generates continuous income, which is bummer for developers.
Actual developers rarely see ongoing revenue from their past works. It's the business who sees the ongoing revenue, because odds are high that any individual developers have moved on with their career, and have been replaced by someone new to kick in 2-3 years of labour.
• Offline-first support with real-time collaboration
• Real-time collaboration with local devices over Bluetooth/ad-hoc Wi-Fi
• End-to-end encrypted real-time collaboration without the server having access to any of your content
• Transport-agnostic sync: use Dropbox, iCloud, and Bluetooth all at the same time with no consistency issues
• The ability to switch to a different cloud provider with zero friction, and to grab your (mergeable, collaborative, and versioned) documents from your current cloud provider without conversion to any intermediary format
• Anxiety-free sync: the user can be 100% confident that their changes will never fail to merge, even if they spent a month in the bush editing their documents
These are off the top of my head, but there are many, many others. And they are features. If enough people build software using these tools, people will get used to them and start seeing the big players as annoying and clunky. ("Why can't I make changes to my spreadsheet when I go through a tunnel? Why did I lose all the changes I've been working on over the last hour? What do you mean this .doc file is just a link to a webpage?")
Is there Big Money in it? I don't know (or care), but I'm going to try hard to make sure that any software I write on the side follows these principles, and I hope others start to do the same.
You could have easily said that "the reason that software is online is a business one" about time sharing versus personal computing, and yet here we are. Focus on the user instead of your bottom line and you will (eventually) win.
I welcome your thoughts on swarmdb http://github.com/gritzko/ron-cxx
That is syncable RocksDB with CRDTs inside. Pre-alpha.
They are definitely in the privacy space more than others, but it shows it's possible.
An app that comes to mind is Omnifocus (iOS/macOS): https://www.omnigroup.com/omnifocus
The other reason is platform control: there's continuous tension between the desire of each OS vendor to differentiate their platform and make applications part of the desktop experience, and the desire of application vendors to be able to deliver to any customer using the minimum number of platforms.
SaaS vendors will build mobile apps (1-2 platforms), and desktop applications in Electron (kind-of 1 platform) as service clients, but I agree that they are not incentivized to build local-first applications.
Local-first software is powered by CRDTs.
If you want to learn more about CRDTs, check out:
- https://github.com/automerge/automerge (author's project, legit)
- https://www.youtube.com/watch?v=yCcWpzY8dIA (deep technical talk)
- https://gun.eco/distributed/matters.html (my Cartoon Explainer)
<3 thanks so much for your amazing contributions to OSS.
CRDT algorithms have documents which grow without bound. With OT, the documents are always minimal and it’s easy to reason about (and implement) trimming operations.
CRDTs are a better tool for distributed applications, but for server client stuff OT works fine.
Runs in production, having done 1TB p2p data in a day, on $99 hardware! Internet Archive and others use it.
I do agree tho, most CRDT implementations have just as many scaling and compaction problems as any append-only log system.
OT is worthwhile to understand.
var a = {b: 1, c: {d: 2}, e: 3, f: {g: 4}};
a.c.z = a;
Every object has its own UUID, which makes circular references & sub-objects easy to reference.
a.c = (UUID pointer to c)
a.c.z = (UUID pointer to a)
a.f = (UUID pointer to f)
Now we can
(a.f.z = a.c) && (a.c = null)
c doesn't actually move, just the pointers on `a` and on f.
c now has a new parent (or could have 2 parents, since any graph is allowed).
Since everything is represented on disk/wire as a flat graph, all updates can be well defined as operations on `UUID.property = primitive/pointer` atomic changes in the CRDT.
This means there are only 7 operations to commute: (I hate switch statements, but someone helping me formalize the CRDT wanted it expressed this way)
https://jsbin.com/hedeqoxusa/edit?js,console (click run to see each operation applied)
You see order-of-operation doesn't matter.
Once merge has happened, history does not need to be preserved (no unbounded growth!) but it can if you want log/history.
Merged states can be stored as a flat graph on disk with a radix tree which allows for O(1) lookups on UUID+property pairs regardless of graph size.
There are some caveats though, of course:
Strongly Eventually Consistent, so don't use for banking.
Counter operations still grow-only but can be done in 12 lines ontop (see https://gun.eco/docs/Counter ). Rich text also grow-only.
Happy to expand on anything else, too!