Offline is just online with extreme latency
blog.jim-nielsen.com
blog.jim-nielsen.com
For example, say you have a form. If you built it thinking online first you'll probably have some pessimistic UI which shows a spinner and waits for the server to respond with ok/error. You can't simply think, okay since we're offline, more latency -> show spinner for longer. You have to re-architect things so that the UI is optimistic, commits to a local database and that local database is synced up to the server when you come online.
In my experience optimistic UI is way more complex to build. Many times the complexity is worth it though.
An offline capable application that sometimes tries to sync can be called "Online with Extreme Latency", but that is not what true ~~Scotsman~~ offline is.
Offline/online is first and foremost about data locality and data ownership. In an online application (regardless of latency!) the source of truth is "the server". Offline application itself is the source of truth and "the server" is a slave.
The OP seems to be talking about thin vs fat clients. Fat client is still a client - it fetches data. Offline application is the data source for "the server", it's the server that has to adapt to local changes not the other way around. Naturally, this is problematic since now you have multiple sources of truth. However, shifting source of truth to "the server" does create online application with extreme latency - fat client.
The longer you're disconnected, the more things diverge, the better your merging/resolution schemes/algorithms need to be.
There are a lot of ways of thinking about and approaching the problem, some of which work worse or better under different circumstances. You see it in distributed consensus, multi-master systems, version control systems, etc.
There's ongoing work in the form of things like operational transforms and CRDTs, but clearly their needs to be a lot more progress in the area.
When you are developing with an always-on connection and a high-performing device, it's easy to ignore all the scenarios where someone doesn't have either one.
But I think it’s part of webTooling in the browser
Cryptographic verifiability make data locality into an optimization problem not a trust problem: https://fireproof.storage/documentation/how-the-database-eng...
Decentralized essentially means absence of built-in omniscient entity - each end every connection is treated as one between equally correct peers. It does not mean that a pair of entities cannot establish hierarchy, it simply means they do not have to.
Take git for example. There is nothing forcing us to treat github/gitlab/etc as the omniscient node even if it is central point of data exchange - you are free to disagree with the upstream changes. Local data is always there, all local overwrites are explicit and so on. Contrast this with something like online game. The server will tell the client what their state is - there is a central entity controlling the truth.
Web can be interesting in this context, since there are multiple "tiers" of nodes: user applications, backend servers, databases. Data control and conflict resolution properties usually are different among nodes in the same tier and between tiers.
If offline is online with with extreme latency, then "extreme latency" must include "infinitely long latency". But accommodating infinitely long latency requires a different approach than if you assume that a server will be contacted at some point.
In the yard you have extreme latency. Sooner or later you'll get thirsty and you'll walk into WiFi distance.
In a national park there's no connectivity. You won't be able to get or receive anything until long after the moment has passed. None of the answers you get will be relevant anymore for most people (what percent of people who visit a national park visit the same one over and over again?).
Then I realized screen contrast wasn't 'there' yet so I shelved it, got busy, and there it remains collecting dust.
not necessarily if the data is stored in a proprietary format.
If I’m running a business and my internet goes out, there’s a huge selling point to a local DB that allows my operations to keep chugging along until Comcast figures out their mess. Internet returns, my DB syncs with a server offsite, and now I have that backup.
You can have a little checkmark to indicate that it’s synced, like many chat apps do.
> In my experience optimistic UI is way more complex to build. Many times the complexity is worth it though.
Yeah, can attest to this. I’m working on an app[1] that strikes the trifecta: p2p, real-time and offline first. All of those things combined makes the amount of tooling, design patterns and resources available shrink to a tiny sliver compared to a typical web-based tech stack. I have researched probably 100 projects that sound promising but almost all have been a poor fit for one reason or another. I opted to build almost everything from scratch.
Kudos to the JS ecosystem. They are way ahead in this space, with rxdb, watermelon, yjs, automerge etc. Unfortunately I couldn’t use any of them because I use a split-language stack.
[1]: https://payload.app/
rotate CRUD mindset to CQRS, now you have a start event and an end event, this is the extent of the pending sync state which can be afforded to the user at any granularity - document level, message level, form or field level, etc.
The cost is that other view queries don't reflect the pending event, only the view that issued the command has an association to the event. Which is usually the UX you want. Consider a master/detail form app where you insert a new record into a collection, but the view is sorted/filtered/paginated and your new record does not match the criteria and therefore vanishes. That is never the right UX. A less surprising alternative UX is to limit record creation to a specific form for the business rules of creation, and that form exists in the context of a specific collection view with business rules around newly created entities of that kind, with an invariant that newly created entities will always appear at the top or bottom of that collection (e.g. new messages are always added to the bottom of the chat history view). Now the optimistic update bypasses the database and simply writes through to the view optimistically and then when the ack is eventually received it seamlessly changes state without any UI jank.
Now you can send many new messages rapidly and if the nth message fails and the rest of the messages went through you get a properly located error. With a CRUD mindset, probably the form is simply disabled until each individual insert succeeds - which means your chat app cannot keep up with you typing!
E.g. if the user is a realtor, then she can't tell the customer "you have now bought the house" if there is no online connection and you're waiting for a sync. You can fill out and upload the form async, but commitment must be online.
You cannot tell the customer "you might have (or might not have) bought the house, we will only know later".
Sure you can - in fact, that's what realtors call offers. You place an offer, and you only find out later if you bought the house, often days later.
Eventually you do need to be able to commit some sort of transaction and get confirmation from the counterparty.
Ah, yes, "in theory, the theory and the practice are the same". Unfortunately, in practice, the theory and the practice are not the same, and those cases are regularly not solved with optimistic UI (even though they should), so there is difference.
I don't find the concept "optimistic UI" to be more useful for talking about it or the implementation by Apollo to be more elegant. It's fine but it doesn't solve any problem except one that was created by Apollo's binding of server data to the client, React-style.
I recently got into using a GPS sports watch. It is the kind of thing I would want to use in an offline fashion, i.e. go somewhere off the grid and track my hikes or bike rides. These devices are designed to function offline for a stretch of time, but they have a requirement to eventually sync to an online system. They will eventually fill up with recorded sensor data and you want to offload that somewhere else before clearing local device storage. More importantly, satellite positioning receivers need a cached "ephemeris" file that helps them predict which satellites will be overhead at a given time to operate efficiently, accurately, and quickly.
Unfortunately, the manufacturers have been infected with smartwatch expectations. They started designing it as if it is always online, and the syncing functions are implicit background behaviors. When doing sporadic syncs and going back offline, it is hard to influence it to "get latest ephemeris" when you known you have internet connectivity and will be going offline again. Worse, these results are localized and it implicitly gets ephemeris for its current location. The UI doesn't allow you to indicate a destination and pre-load the right data to allow fully offline function on arrival.
Interesting, I've never heard of that particular optimization for GNSS before. I know GPS transmits ephemeris information in each frame, since that's the input data for the positioning calculations. I've got a number of Garmin watches, and they've always been able to get a position fix even after being disconnected for weeks.
I find GNSS implementations very interesting, which manufacturer is making their watches like this?
Their documentation states that this may expire after approximately 30 days or if you travel more than 200 miles, while it will update during syncing.
I've seen it expire in less than two weeks with daily use of GPS but phone syncing disabled. It will still get position, but it can be the difference between an almost immediate fix after opening an activity menu or a delay for tens of seconds to minutes. Distance and pace measurements also seem to be lower quality when operating without a current ephemeris file.
Isn't that exactly what the article argues for though?
> This kind of idea would move you away from a product full of API calls to one based on data synchronization.
I'm not sure you got the point. The design guideline that "Offline Is Just Online with Extreme Latency" already reflects very specific architectural requirements, and state transitions in the application life cycle. We're talking event-driven architectures, batching events, flushing events in offline/online transitions or even when minimizing windows, pulling events when going online, etc etc etc.
I'd go even further and claim that this whole "pessimistic vs optimistic UI" thing is just "adequate vs broken UI design",regardless whether the app is even expected to go online.
I've also worked on systems like that where a mobile user could loose connection at any time and they are indeed very complex to get right.
I'm not sure there's a perfect way to do it but we ended having have certain functions that had to be done online and others where the user built a "request" for service that was handled optimistically with failures sent to the user's inbox later.
I think that optimistic UI isn't more complex to build: it's just less familiar.
In a broader sense, UI/UX design culture places way too much focus and value on familiarity.
The best UI/UX designs I have ever experienced are when software design diverges from the norm and tries a new approach.
Having multiple records of state that have to be resolved when systems come online is complex. Anything requiring consensus is difficult to do.
In an offline-first experience, you need to be very careful that you treat everything like a source of truth. You also need to deal with schema migrations and business logic migrations, since you need those to partially live on the client.
Now of course I was judged on uptime of my app that relied on it.
Finally just cloned the data to a local MySQL and refreshed data frequently. Database team totally clueless that app servers were hosting their own databases.
extreme latency for any new data fetch/processing (an http roundtrip or more for each interaction with your data)
someone else being in control of the programs you use, forcing changes and updates whenever they like, renting them as a service,
your data hostage, snooped at, sold to advertisers
a limited UI experience based on the DOM and browser APIs
crappy OS integration
(often) ads
There's a great deal of technically unnecessary complexity that many people here handwave away because "that's just how it is."
It's kind of funny how, e.g. above, the emphasis is on "UI," when that's not it at all. It's the underlying infrastructure that's often encrusted layers of unnecessary (but profitable) stupid.
Anyone who thinks UI is where the latency and complexity resides should do a 3-mo residency at a cloud provider running actual infrastructure--not VMs and the like, but the actual routers, backbone links, service provider peering, and servers. It'll change your perspective on complexity for sure.
This is a much bigger problem than people realize. Instead of working on improving user experience, we've got people explicitly working against user experience by injecting ads into it, for the simple reason that they have your attention and can then sell it. It's the attention economy, and it sucks. Fuck ads.
The trailing part that's left off is "for free". If you want all the content we receive for free today, someone has to pay for it. So either people pay with their attention or their wallet, and consumers have by and large made their choice of ad supported services over paid alternatives.
This one doesn't have much to do with online vs. offline except that you have to be online to see the "latest and greatest" ads. Stale ads can perfectly well be shown offline.
They can, but an Android app I use regularly simply won't show any ads if it cannot access the internet - I get its premium version for free just by pressing a button. Meanwhile I've seen the user-hostile version of that on an iPad years ago, where an app simply locked itself until it was able to download ads.
Since the rise of cloud applications and electron based applications I'm back at writing faster than the application can process my input. And I'm not a fast typist.
If the application you're using (the one(s) you're referring to when you say "cloud applications") is sending the key input to the backend and waits to render it client-side until it received a response, it's doing something horribly wrong and if that's how they write things, you probably want to avoid that for a multitude of reasons. Please name and shame though, so others can avoid it too :)
Same goes for Electron-based applications, there is no reason they'd block showing the text you inputted, and if they just use <input/> without doing anything egregious, you really shouldn't be able to type faster than characters appearing on the screen, something is really, really wrong then, and I don't think it's because of Electron.
I'm 99% sure nobody does that, that would be insane :D.
It's most likely caused by React and the use of controlled input components without thinking about performance. If your input sets it's value from React state, then you're updating the state on every keypress and re-rendering the component. In the most simple case, it's probably fine. But depending on your component hierarchy, it's pretty easy to end up in situation with over 50ms of typing latency.
I imagine you haven't seen how people implement autocomplete on web forms ;).
I agree, hence the surprise!
Although parent explicitly calls out "cloud applications" so I'm guessing online has something to do with it.
And in the context of online/offline, it wouldn't matter if it's internet connected or not, using controlled input components with state changes going through a deep hierarchy would introduce that sort of delay no matter if you're online or offline.
MS Teams does this when starting a new chat - you select the recipient and start typing and Teams will do two unexpected things shortly afterwards:
1) Clear the text you typed, and
2) Reset the cursor to the start of the line[0] https://github.com/microsoft/vscode/issues/161622#issuecomme...
On desktop applications they may lack in the response on the screen if they or the computer are to slow, but with cloud apps sometimes keystrokes go missing and the entered data is in the wrong field.
The operating system recognises all keystrokes, but the browser seems to skip some.
For example, I use Apple Music every time I leave my apartment. Put my headphones on and start some music. All my songs are downloaded locally so it should be fine, however, as soon as I close my front door until after I step off the elevator, no music will even attempt to play. That’s because Apple Music is blocking itself on some sort of network call/drm check because it thinks I’m online and it should work.
If I go into airplane mode and hit play, everything works perfectly.
In the nitty gritty they are the same thing, but the way you say it changes the way you picture it and what parts you make sync vs async
> Unfortunately, some of the open technologies for building more local-first, p2p applications just aren’t there yet. Like Peter’s point on webRTC.
I guffawed at this. I mean, seriously. In the days before the internet was widespread, like mid-90s, there were actually multi-player games that worked over LANs, both IP and IPX. Somehow, people were able to configure multiple computers to talk to each other with this funny numbers called "addresses" or something. DNS existed of course, but you could also add to /etc/hosts, Novell had a thing, everyone had a thing.
The problem is that somewhere around the mid 2000s or early 2010s we forgot that anything networking-related could be based around non-web protocols and stacks. What a disservice we've done ourselves.
Not that I'm advocating people type in IP addresses or have to set up their own DNS servers or admin their own LANs, but wow, it's like part of our collective brains is just missing.
People always talk about “how expensive” it is to send packets to Oregon to turn on your lightbulb in seattle. But the cost is already fully loaded and paid for. If you didn’t notice it, maybe that’s because it’s actually not more expensive? Yes latencies are higher, and that’s literally the only downside I can imagine. But it is a lot more reliable, flexible, simpler to deploy in any environment, robust.
You make points below that are good, but let's be clear. You're talking about simplicity of interop and configuration that's gotten better, rather than simplicity of the software stack. There are so many variables that have changed by adopting the web stack that I don't think any other dimension of simplicity has been achieved.
If we had instead focused networking protocol design on the simplicity of configuration, discovery, and naming, then we'd have been better off. It's not like you need an electron stack, an apache stack, and a cloudy (planet megascale!) datastore for that.
I also think it’s kinda not true that protocols have stayed stationary in that time. Certainly Bluetooth has improved, there’s lot do innovation on small device adhoc networks, there even a lot of innovation on internet protocols over the last 5 or so years. My observation (having been at Netscape at the time) was the real demise of network and internet protocols until late was the MSFT destruction of Netscape. We were a huge driver of protocol innovation and had demonstrated there’s gold in them thar networks. We hired a lot of protocol and standard developers and built products that implemented as close as our talent allowed to the spec. It was a heady time. Microsoft realized at the time this push really threatened their core business model. They so publicly and roundly pwn’ed us out of existence that it created a massive chill in anything internet tech related (most of us went into e-commerce as a result). It wasn’t until sort of recently Google self servingly started bullying standards out the door that things sort of unwedged again and despite the kind of sad way Google primed the pump, things are really exciting right now on the internet.
I’ll note that our Mozilla play was a really important effort on our part to ensure that culture we had built around open internet standards and technologies could stay afloat into the present. I’ve been happy to watch Gecko and Xul and others grow from their sapling state to where it is now. It also prevented Microsoft from establishing a real hegemony in internet standards. But most of all, I am so happy to see a real C/C++ replacement candidate emerge (Rust) from that effort. It was a real thumb in Microsoft’s eye to one up their $1B free browser anticompetitive play with taking our then worthless IP and establishing a long lived foundation around it.
In the scheme of things very few people were able to set this up. It's definitely unfortunate that so much software no longer works without a server component you dont control, but one of the major reasons for it is that the typical user can now do it because they don't have to set up the networking or understand much about it. A little later software companies realised they'd actually rather keep control and stopped distributing the server component at all.
WebRTC is the focus because it won by a huge margin, other p2p protocols are a footnote. Everything uses WebRTC whether it actually has a web client or not.
Remember when there was no "cloud" and just locally executing software with local persistence?
I want very much to return to that. There are a few use cases for me, personally, where SaaS / "the cloud" makes sense but for 99% of what I use a computer for, I want local only. I don't want my application to require an active Internet connection. I don't want it sending data to someone else's computer. Probably most importantly, I don't want the software that I pay for to change on me without my direct opt-in.
The fact that we have articles about offline-first, cloud is "optional" is hilarious when this was just normal every day software only a few short years ago. I can't stand how trend-happy our industry is. "The Cloud" and "SaaS" solved a few problems for a few people ... but we went all in as if there were no alternatives and those of us who don't require mobility and would prefer that we can choose when to "upgrade" and what data to "share" were left behind.
To provide a counterpoint, though, during that time my entire family shared one desktop PC. Looking around me, right now: work macbook, personal linux laptop, smartphone, steam deck. I suspect that the workflows that were "fine" in 1995 would really wear on me today. Especially the ones that involve migrating documents from place-to-place for collaboration while trying to maintain a canonical master copy somehow.
Today, because of de-facto reliance on the cloud, "setting up" a new machine - regardless of its OS - takes me about 20 minutes. If my laptop fell off of my bike, that would suck, but I wouldn't irretrievably lose important data.
There are downsides to the current cloud-first paradigm too, of course. But I don't think it's _all_ downside.
I never migrated the majority of what I do to "the cloud" and I have multiple devices, laptops etc. in my house. Setting up a new Linux install takes me about 20 minutes. Wouldn't know about Windows. Then again, I probably use way fewer "services" / "apps" than most people. It's funny, I've always been a very tech savvy computer nerd. I've developed software for a living for 25 years. But the older I get and the more the industry changes the less I find I use "modern tech" as a consumer.
I'm also not thinking back to 1995. This all started, in my memory, following the "mobile revolution" at the tail-end of the 00s. That's when more and more software that I use every day stopped selling perpetual licenses and started charging monthly subscriptions. Everything went "mobile first" and "cloud first" and it got more and more persistent as the 2010s went on.
On a positive note, there is some software that used to be inaccessible due to price, like Avid's Pro Tools, that I can now afford thanks to the switch in pricing models.
But even if it took me an entire weekend to set up a new device, I think that I would still prefer that over needing literally everything to have Internet access. It removes the control from me, the user, and places it in the hands of "the corporation." They can change the software without my consent. They can suffer outages (to be fair we can have local hardware failures too but it's under my control which makes a difference). I remember almost returning my PS4 when it required me to connect it to the Internet just to be able to use it on first boot.
It may be a "get out of my lawn!!" reaction, but I find modern software distasteful.
I want to use software that enable me to invent and do nice things. Not software that locks me in a pre-designed process. Even if most of the time I don't go inventing, and if the pre-designed process is nice, that difference still sores me.
Investing time and energy into a tool that can be ruined or lost at any point in the future, or might turn out to be too limited, is a huge risk. It doesn't take getting burned too many times to start getting a lot more wary.
I suspect preferences will hinge on peoples' budgets for personal responsibility. As my non-digital responsibilities have increased, I've found it nice to be able to delegate to "the cloud" - even at the loss of independence & control.
If there were a personally-owned "cloud" setup, I would prefer that. A box that plugs into my fiber connection and provides the equivalents of the cloud services I use, with data stored locally and backed up automatically to a secure server. A man can dream.
Alas, that's a pipe dream because of the way things work now, and I don't necessarily want to inflict a luddite lifestyle on my wife and children. But the allure is still there to setup some crackpot scheme that will keep me from idly pulling out my phone to check whatever just because my brain is idle for 5 seconds.
Bonus points for the irony of LotusScript being ECMAScript, which is also JavaScript.
Oh, and Notes generating the native app as a web app natively was pretty fluttery.
I have to stop, because I can no longer remember why people were switching away from Notes so furiously. And there was probably a good reason like IIS.
NotesSQL was a great little ODBC driver to make relational db calls from the NoSQL backend of Notes. Many folks owe their sharpness in the universal standards of ANSI SQL to get data ported to MySQL.
Edit:
Remember it? That's still my reality, because for almost everything I do the cloud and SaaS are not good solutions.
Absolute rubbish. The proliferation of SaaS has been absolutely huge for business productivity, in fact, I would argue that my professional services business doesn't exist at anywhere near its profitability without it.
> I don't want my application to require an active Internet connection. I don't want it sending data to someone else's computer. Probably most importantly, I don't want the software that I pay for to change on me without my direct opt-in.
What software specifically? How to collaborate on literally anything business wise without a internet connection?! Now, is there a case for offline software? ABSOLUTELY. But to call this whole thing a fad and a trend is absurdly disingenuous.
EDIT:
Just saw your reply below:
I have multiple devices, laptops etc. in my house. Setting up a new Linux install takes me about 20 minutes. Wouldn't know about Windows. Then again, I probably use way fewer "services" / "apps" than most people. It's funny, I've always been a very tech savvy computer nerd. I've developed software for a living for 25 years. But the older I get and the more the industry changes the less I find I use "modern tech" as a consumer.
Ahhh makes more sense, you're a developer. I get that YOU might not need SaaS tools, but this is a very myopic take.
But you need to understand that even as a self-employed business owner (product company, not contractor btw) for 15 years, I didn't take advantage of many SaaS tools at all.
Collaboration is niche. B2B productivity tools are a big industry (ironically I actually work for a company that makes a popular B2B collaborative SaaS product) but in the global scheme of things it's still a narrow niche when you consider the broad field of computing and software in general. Not everyone needs to be mobile, not everyone needs to collaborate. There are loads of software applications that can be local only, but there are few options because of the way our industry works.
And your post is timely because I was just thinking to myself this morning about how the biases that are introduced by Product Owners who come up with these ideas in work-related contexts, even if they are not the POs of B2B or "collaborative" software, still permeate non-B2B, non-"collaborative" software. They are in "work mode" and so they tend to think in terms of "How do I interact with this software. What do I need?" rather than actually considering the use cases for their every day users.
Take gaming, for example. Multi-player gaming is quite popular and those who seek it out ought to have options there. BUT ... I stopped gaming in large part because the industry went all in on online multiplayer and single player gaming options became more difficult to find. It's not that they don't exist anymore, it's just that I don't want a console that requires an Internet connection to play a story-based game like The Last of Us.
It's the following that pisses me off:
- The "all or nothing" mentality of our industry
- The ability for companies to change software that I use and pay for without me being able to decide if it's worth "upgrading" or not
- The lack of choices
I think it's pretty rich that we're both accusing each other of the same thing. You made the argument that because collaborative B2B tools are useful in a business context that it is "myopic" of me to want something that doesn't cater to that niche. I can make the exact same argument right back at you.
> You realize I was speaking for myself, right?
So "us" is just you? Your "us" is a very small minority, which is the point of the people who are responding to you.
> solved a few problems for a few people
Are you trolling?
“How do I share this with other people (who don’t pay for the software)?”
“Can we work collaboratively on this?”
Etc etc.
People are used to things being in the cloud already. And I am not talking about tech people.
Once you’ve experienced live collab, there is just no turning back…
>> People are used to things being in the cloud already. And I am not talking about tech people.
Most people use their phones for almost everything, and this is almost all local execution, local persistence and huge amounts of cloud-based collaboration. I wish desktops were on this trajectory.
I work on my documents on my computer, my phone and my iPad.
My IDE is hosted in AWS Cloud 9 most of the time because I travel a lot and while my internet may be spotty, it’s good enough to connect to the hosted environment and I get full speed internet between my IDE and the Internet when I need to download something to it (packages, Docker containers, etc)
My photos and videos automatically get backed up to iCloud, Google Drive, Amazon Drive and OneDrive.
If I dropped my phone into the ocean, I can go to the Apple Store, buy a new phone and it’s just like having my old one.
I’m sympathetic to your point but most apps I regularly use require internet anyways.
There is much chat about real-time collaborative apps, but the reality is that most people are working async most of the time. If you start with the concept of offline first, with eventual consistency for collaborative work, you implement all of the same conflict resolution as if you are implementing a real-time app with conflict resolution. Then adding real-time collaboration later is easy.
This is where CRDTs (conflict free replicated datatypes) work better than OT (operational transforms). OT need more supervision from the server, the further you diverge from the servers state the more likely there will be a corruption. CRDTs are inherently designed to work with massive diversion and no central authority, really they are offline first.
> In the case of webRTC, I've seen so many projects do this where they’re like, “It‘s peer-to-peer! You just connect to the signaling server and then the things talks!” That’s not really peer-to-peer cause you've got a server. If you've already got a server...my advice is: don't mess around with peer-to-peer stuff, just use that server.
The prevalence of NAT really did a number on ability to have properly p2p communication. And, well, rampant insecurity of default user setups making most internet connection be at the very least behind "only allow outgoing traffic or traffic initiated by user" type of firewall.
Even if you make all that nice code with each user having it nice and secure private key and having to accept someone's public key to start communicating there is still problem with distribution and ascertaining that user is really who they are.
And even if they are, well, they can be compromised, so your local app also have to be very secure or else the potential malware can just go between your contacts and compromise everyone using the app.
Or, you can just use SaaS and don't worry about it, or at the very least not worry about more than one app having ability to compromise your system
And even if you solve those you have an N^2 problem in the naive solution, and to reduce that you need mesh networks which just don’t have the QoS we’ve come to expect (to my meager knowledge).
Imo we should be able to find a better middle ground, because full decentralization is insanely hard. If we had federated standards and tooling like web and email, but for modern use cases, I’d be a happy man.
Despite (or perhaps because) the tech craze over the last decade, the developments of such open and useful standards leaves a lot to be desired. Almost everything cool these days is some app with perhaps an api that is branded as a “platform”. We’re basically treading water.
Now, you need 5Mbit speeds just to collaborate. Funny how the tech "progressed".
What you have in stock is nostalgia, my friend. Everything worked? Like Windows 95 crashing if you looked at it sideways, Linux requiring you to compile drivers for pretty much any device yourself, like fifteen competing printer protocols all incompatible with each other? You might want to get a reality check :)
We used the "Excel Shared Spreadsheet" functionality. The file was hosted on a network drive. It corrupted or crashed so frequently - even when only one person had it open - that we took to copying it to your local machine, making the network copy read-only, editing the local copy, then copying back onto the network drive. Even this was buggy, so possession of a stuffed octopus represented a lock on the file. Microsoft themselves got involved at one point, as we were a reseller for their products. Their solution was no better.
So if there was a halcyon period for collaboration at some point in the past, I'd like to know when it was. When I see 100 people live editing the same Google Sheet, I think perhaps we are in the golden age.
Today, with spotty internet, software you have or could have on your device might or might not work. Just recently I have seen a website which loaded, but was non-functional until 2 mins later a cookie warning was also loaded.
Nothing in there is about
> the number of active internet users, malware, services, goods sold
but about not building for applications only loading half (because we are still waiting for some obscure JS or CSS from not the original domain that noone really never needs.)
Picking a random example: "AirPrint®, Brother iPrint&Scan, Mopria™ , Cortado Workplace, Wi-Fi Direct®".
Will a lists like that be enough to overcome your fear and skip the costly upsell to exactly the same printer, but it also understands PCL6? (likely a bit flipped in the firmware...) You decide. It's like a reverse confidence trick: "look, it's 2023 already and it's awesome, you really don't need PCL6 anymore. Trust me."
you can use gimp to take images and make them into coloring book pages, you can print instructions, calendars, drawings, diagrams; prompts for writing, your own writing to make sure that the "screen" isn't tricking your brain making you miss editing/grammar errors. Coloring pages, maps of countries, and other educational things are more fun if you can hold them and put them on your wall, if my kids are any indication. Having a scanner and a copier and a networked printer in a single box for what they cost - that makes it difficult to see the aversion.
I have had fancy e-readers since the original kindle, and before that a Clie and two palm pilots with acrobat and an epub (or whatever) reader. and i still prefer dead wood. I read more books on paper than e-ink. I read more on a computer screen than books, though.
Now our printers, cars, and soon fridges are DRM-locked and doing surveillance.
And they can stop working at any time if the manufacturer goes bankrupt or simply stop supporting them.
I also run a PBX, just in case coturn decides to stop working from lack of use. Now i gotta make sure i start the PBX app (3cx) once a week otherwise android helpfully removes notifications for me!
AIM, ICQ, etc were great - and i'm discounting "presence", because that was a small part of it. realtime communication for "free". I've been running chat servers for over a decade, and right now, there's three people on my matrix server, me, wife, and my best friend. there's five people on my PBX, same three plus my youngest child without a SIM card, and grandma, so grandma can call kid and vice versa. I've run ircd, matrix, rocket.chat, mattermost, a couple of the jabber systems, ventrilo, teamspeak, hand-rolled garbage.
we've had people sign up and chat once, but as soon as notifications stop working we never see them again. Heck, i have 3 PUSH accounts just to make sure that notifications go through everywhere.
Meanwhile, 20-25 years ago, load up trillian or AIM/ICQ and there was everyone, literally everyone. I know facebook ate that lunch, but i refuse to have any facebook apps on my phone, and most people i know refuse as well, so that's a non-starter. Texting is still spotty, because VoWiFi tends to just magically stop working if you leave the wifi area a couple times in a day, so texts will sit in limbo until you realize. Phone calls are the same way (except via PBX, those always seem to work for some reason...)
running out of IPv4 and the ubiquitousness of CGNAT have killed what the internet was. And that's sad; "nostalgia" isn't necessarily a negative thing. It used to be better, at least in the West. Maybe wechat and signal and telegram and whatever work better for the rest of the world now than any of our stuff from 25 years ago, i wouldn't know, because i know exactly zero people on any of those services - i've checked, once every couple of years.
There are many things that killed what the Internet was in the 20-30 years hence. IPv4 exhaustion is regrettable, but entirely predicted, and for those in the west, not a huge deal. I can pay any number of cloud providers a small fee to host me a box with a publicly routable IPv4 address. CGNAT is annoying, but the number of libraries which will punch a hole in that NAT has also expanded by a huge margin since the 90's. So those up things are unfortunate, but I think there are many other things that changed the Internet, some for the better, some for the worse.
no, because everyone uses something different and there's no "trillian" or meebo for all the shiny-tech. If we're going to play "encrypted chat", that no one can agree what that means, or what service to use, may as well use my own. Hopefully matrix becomes more popular (it probably won't, unfortunately*), but if not, i still won't use the likes of telegram or FB messenger or tiktok DMs or anything of that nature. there's not a good IRC client for phones, last few times i checked over the past 15 years.
* Matrix has the curse of being good enough on the client side, but kind of a PITA on the server side. Synapse, the reference implementation is in python, which is single threaded, so joining a large room "as-is" out-of-the-box is not feasible. You have to split each of the synapse elements into a microservice that gets its own python instance, but the documentation was (and probably still is) quite sparse. I'm sure there's a discord where i can get help.
It's not much better on modern Fedora. The drivers do come precompiled but you have to reboot sometimes twice per week to install updates.
DSL & Cable encapsulates Ethernet. City-wide networks are again Ethernet. In fact, everything is Ethernet with some routers between.
So, you're actually using a mixed-carrier Enthernet LAN with a couple of NATs in the way. You can build another LAN on your LAN with VPN, too.
That was my point responding to OP. We actually progressed and we are facing new challenges.
A country wide router still can stay in service for a decade. Residential FTTx installations consist of a TelCo class switch, maybe a router, compact DSLAMs if you have DSL service and a couple of fibers in, tens of fibers out for FTTH customers.
It's faster hardware on the same stack. Nothing more, nothing less.
...and, oh, link-speed (transparent) DPI equipment in key places for lawful interception.
...and possibly a small DNS server (or a hosts file somewhere easy to get).
That's all you need.
If you want to navigate via IPs only, you can do with a DHCPd.
Depends on what you consider "work from home". You might not be able to chat with your co-workers without a WAN connection (unless you go really old-school and use a telephone), but you can for instance develop software from home without any working network connection (and git makes it easier by allowing you to create commits while offline; see also the "git bundle" subcommand). I've done so occasionally when the Internet connection goes kaputt.
They're using SQLite on device with a sync service online. Their team have credentials that give me confidence: https://electric-sql.com/about/team
Surely google maps knows I live in NYC by now but if I’m between stations on the subway the odds of me being able to pan around at various zoom levels is surprisingly low. Surely basic semi-static highway and geographic information could be locally stored (this basic information used to fit in a spiral bound book we’d keep in the back seats of our cars).
iMessage just gives up if a message can’t go out rather than queuing it.
Heck most weather apps won’t even show me the previously downloaded forecast if there’s no network, they just show “-“ or a sad cloud image.
I assume they still have the "offline maps" feature, but you set it up and select regions by yourself. I have no idea why they haven't automated that using their location tracking and AI features.
Re: iMessage Yes that's a shame. Only excuse I can imagine is there should be some timeout by how much a message is allowed to be delayed. And because of SMS delivery etc this would probably cause a lot of complexity (cellular has its own queueing system, at least here, and it's unreliable)
They have had a few meetups so far which have been really good (including PVH speaking at the first one). And many of the local-first builders are hanging in their Discord.
This is literally what was pioneered by Lotus Notes 32 years ago. When everything was local, their rich-content databases could work just fine locally and also replicate on any desired schedule with other copies of the same databases/apps. The model was explicitly Seldom-Connected, so you could run stand-alone in a remote location for weeks and send/receive updates whenever you got to a phone line, or could have it collecting/distributing updates every N seconds on a persistent connection.
A vastly superior model to everything I've seen since, and it is too bad that they, and especially IBM blew it with the weird programmability models and corporate-extract-every-dollar approach.
Acting as if this is a somehow new concept is just a display of cluelessness, but it is good to see good concepts getting some attention (again).
Well, because it's distributed computing. Distributed computing is hard... really hard! Especially for applications requiring any level of real-time-ish interactivity between different users.
Two computers talking to each other in a room via a server in Virginia is not absurd. The physical distance is abstracted over completely, the application engineer needn't think about that at all, that's the beauty and point of protocols like tcp/ip.
What cannot be abstracted over is the single source of truth in the networked system (the server) that makes a whole class of really hard problems go away. So as much as you can describe the remote server as a problem, because of latency perhaps, it is a vastly lesser problem than the distributed computing ones that'd be introduced in the alternative!
That is the basic reason why things are the way they are, it's really not absurd or surprising at all, in my opinion.
Notably:
1. product people and interaction designers don't understand or care about the difference, which isn't strictly their fault, because
2. users don't understand (heck some even want a team of engineers making sure the DB doesn't drop their writes all day every day) and only care when the software can reasonably be used offline anyway, which means
3. standards, protocols, and platforms have baked in dependence on servers (oauth, tls, ip, mobile device native apis, the goddamned entire browser, etc.) for example, you can't do anything on iOS in the background unless you have a server sending the devices silent push notifications every 5 seconds ... also
4. true offline with anything more than 1 client is a distributed system and instantly hard, so
5. the entire tech community is not trained to build offline applications, we're trained to glue react UIs onto CRUD backends because thats how you 10x your SAAS revenue in valuation over 3 years.
I am lucky enough to have worked on a few products where we did have to care about offline and it still ended up being relegated to the realm of "obscure edge case", for all intents and purposes. So you end up continually justifying why you're spending resources on supporting these weirdos who don't have reliable internet connections.
Sigh.
/rant
These days, I do honestly think that some things just need to be online. The internet is meant to be a globally addressable network. You're not supposed to have these little islands of accidentally private because-we-ran-out-of-addresses-and-hacked-the-shit-out-of-the-protocol-(NAT) networks. You're not supposed to be afraid of someone sending port 22 on your laptop an ssh handshake because the crypto is good. The whole IP world is supposed to be zero trust. We dropped IPv6 scopes because they were a crazy idea for a different world where we all ran federated networks and whatever.
The real point of "local first" in our modern world is about data ownership. If we can do anything as a community to help move the needle and get to a better place, it's building applications where clients truly own the data and the server just provides convenience and redundancy. Give clients cryptographically strong identities. Use webauthn. Sign requests. Trust users with their own data. Help users be sovereign and we solve lots of privacy and centralization issues all at once.
/soapbox
For the local first part, witch means "casually connected systems", the extra layer of complexity offer another layer of control and resilience: an autonomous car, capable of limited inertial navigation, with full local maps, routing capabilities etc can move even without GPS. Sure, it's more costly and complex than a connected-only car, but it's FAR safer to use.
Similarly an autonomous ship is far safer than a connected only one, for obvious reason.
Similarly in human terms local people with local power are FAR more resilient and effective than meat-based dysfunctional robots tied to some procedure. A VERY small simple example: I'm moved to France from Italy some years ago, now I decide to convert my perfectly valid Italian driving license in a local one. There is an automated (long) procedure. When I start it I've have had to give a scan of my Italian driving license witch happen to be expired and so got renewed BEFORE the procedure complete. In a resilient system I simply can upload the new one, sure this act trigger some extra actions, but extra actions who will happen anyway, only just after a bigger delay. Did you remember the Toyoda model vs the Ford one? Well, local first it's the same. At any scale.
e.g. physical books can last centuries, an online blog - not so much.
I just developed a local-first read later software using CRDT technology. aka
1. all data is stored locally, one page corresponds to one CRDT file. CRDT file is a single source of information
2. use sqlite as cache for query and full text search.
3. use the version of CRDT as the file name. It is guaranteed that there will be no duplicate file names and never file content changes. Only files are deleted and added.
In this architecture
Users could synchronize their data with any intermediate media. Such as hard disk, iCloud, http interface, webdav. It is guaranteed that there will always be no conflicts.
Becomes lot more harder when you have some 100 agents in the field, taking orders on their mobile devices and you have to do full double entry book keeping even when offline.
It is not suitable for processing orders.
When the document is loaded into memory, it has a unique uuid as the agent id.
If the current version is
hash(a1 b2 c3)
If two people edit the document at the same time the name will become
hash(a1 b2 c3 e4)
hash(a1 b2 c3 d4 d5)
After merging, the document name will become
hash(a1 b2 c3 d4 e4 d5)
a1, b2 is "Lamport timestamp" https://en.wikipedia.org/wiki/Lamport_timestamp
the full version (a1 b2 c3 e4) is “vector clock” https://en.wikipedia.org/wiki/Vector_clock
say (a1 b2 c3) is "abc" -- (a1 b2 c3 d4 d5) is "abxx" and (a1 b2 c3 e4) is "aby"
what is (a1 b2 c3 d4 e4)? it isn't "abxxy" or "abxy" or "abxxy" or anything else, because d4 and e4 are concurrent/conflicting updates, which have no single well-defined resolution (without additional information)
meaning, this is a conflict
if you just say that this resolves to (a1 b2 c3 e4) then this is LWW but you've lost the information in d4 and d5, so whoever was agent d has written information that was good for a little while, but then ultimately lost after merge, right? so this isn't consistent in any useful way?
After folks explained that this could require a multi-day, multi-tens-of-thousands-of-dollars journey for the customer, we re-architected parts of the system to be tolerant of offline failures but still resilient and secure and saved the customer a potential round trip back to the field.
1. Cloud is not cheap, so doing a proper DR in a loud environment is going to price out small/medium businesses.
2. Most customers don't have staff skilled in the realms of "what if" disasters, and have not tooled to spin up a copy quickly in a new region/provider should their site get nuked.
3. Most cloud customers have zero plan for what to do when cutoff from their cloud providers, because of above.
-
It would be cool to pay for "ghost infrastructure" (I dont know if this exists already)
But... the idea would be that as you build out your VPC/whatever environment, the cloud provider creates a ghost version of that infra in every other reegion (meaning all the configs and everything are mirroed to a repo at the other locations and allow one to spin up a complete copy of primary infa wherever one likes, and youre basically paying a small amount for a clean, up-todate DR copy of the environment as "a cost of doing business" but in the event you needto, load a snapshot of everything (sans actual data - thats extra) to just launch the environment...
We built this manually in the past (manually meaning the ability to launch an instant VPC and migrate traffic to it... but this was a few years back and I am sure that its far more mature in the real world now...
So what options do people have today?
Maybe a simpler way to put it - if what you are building requires client/server for security guarantees, cloud is IMO an easier delivery model than forcing your customer run their own database. Cloud is infinitely better operationally for these users.
If they want to control the versioning, then on-prem is the answer.
If permission modelling is no issue, offline first plus sync is optimal I think.
It's just easier to switch to a different tool like IM or phone when one cloud tool is down so you can quickly discus the topic there, that to postpone your mental train for when everything is eventual consistent. Or to switch to the 'offline' tools we already have, like e-mail, which require you to rethink your train of though to the different medium.
Also central authorization and authentication are not to be overlooked. Those problems also suffer heavily when partitioned.
There are a few ways to do that using those tools, and it's very easy to implement. I thought it was great, especially using a locally installed CouchDB, but my users didn't really care, and I mean not a bit.
I don't have any big corporate users but there are some pretty big businesses and I could not convince them to run a DB on their side. They did not want to fuss with it.
Some of them have been using my apps for over a decade and there's not been any serious issues of downtime so there is no sense of urgency even though I've made it clear that can happen, and the importance of having their data onsite, in their possession.
I use it for my own accounts on my apps. CouchDB makes that very easy to do by itself. It's a built-in feature.
However stuff like matrix works fine on starlink, because it will eventually be able to phone home and do all of the queued transfers, and that's fine.
There are things you can never do on your laptop. Offline is limiting, in many ways. You know how I know? I grew up in the 90s. Online is a magical godsend of capability and functionality. Offline is cute. Online is what we dreamed of.
Should you have offline modes? Of course. Should you have some offline-first applications? Of course. Is offline just online with extreme latency? Of course it isn't. Use your brain. Think.
Or that offline is zero network latency for the stuff you have on your computer.
Or that offline is, when functional, like online but without surrendering all control over your computing to someone else.
All these metaphors have their own limited scope where they are true and most appropriate.
I've considered Git but not sure how much overhead it adds.
Effectively sync systems like DropBox and iCloud do this by flagging conflicting edits when you come back online and duplicate files (new a new file name) so there is no data loss.
For a lower level file system though that's not really possible as you may be running databases, or other data models, that need to understand conflict resolution themselves.
If you've considered git, I would suggest looking at git-annex (https://git-annex.branchable.com/), which is tuned for that kind of mostly-offline use case.
https://en.wikiquote.org/wiki/Andrew_S._Tanenbaum
The latency, however ...
A European station wagon with the rear seats folded takes 1555 liters, with 95% utility that is 6422 tapes. That wagon can hold 289 Petabyte of compressed data
From Amsterdam to Rotterdam, it takes about an hour to get your data to its destination. That is roughly 640 Terabits per second. For comparison, AMS-IX peaked at 11 Terabits per second last year.
Then only if and when I have a good pressing reason to give it a GUI, or make it client/server, or require connectivity, or be cloud-hosted, do I carry out a refactor/redesign to add those features. And even after I apply the refactor/redesign I strive hard to maintain a UX configuration which retains all those original default qualities. Maintaining them side by side with the added alternate modes.
Maybe its an age/era thing. Because I'm increasingly surprised when folks DONT use this approach. Felt like a design no-brainer.
The private term I've coined for this philosophy is CLIFMO.
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.In fact, ad-hoc random person-to-person communications online are gatekept by "the algorithm" which would rather answer your question directly to keep you from meeting another warm body online.
sounds like
"Linux is a semi-truck with 20 gears to operate. Windows is more like a car. TempleOS is a motorbike. If you lean over too far, you'll fall off. Don't do that."
offline means no expectation of a response, that is very different than an extremely delayed response
If it works locally and there's no need for syncing or communication in the first place then yeah, problem solved!