Engineering for Slow Internet
brr.fyi
brr.fyi
I feel like some devs need to time-travel back to 2005 or something and develop for that era in order to learn how to build things nimbly. In deficit of time travel, if people could just learn to open web tools and use its throttling tool: turn it to 3g, and see if their webapp is resilient. Please!
The problem is the CCP.
The only fix is for the people to rise up against them.
This doesn't even have to be violent. Most of the former Soviet Block governments fell without any bloodshed.
What's the alternative? Wait for Xi to "make his mark on history" in the same way that Putin is doing in Ukraine because it's "naive and dismissive" to even talk about unseating him?
That was Gorbachev. Most leaders of any country would roll tanks.
If it was all that easy, it would have been done a long time ago.
I was born and live in the ex-Soviet country, and stating that Soviet governments fell without any bloodshed is a proof of ignorance.
Anyway, this is very unrelated, but I'm in the USA and have been trying to sign up for the official learning center for CAXA 3D Solid Modeling (I believe it's the same program as IronCAD, but CAXA 3D in China seems to have 1000x more educational videos and Training on the software) and I can't for the life of me figure out how to get the WeChat/SMS login system they use to work to be able to access the training videos. Is it just impossible for a USA phone number to receive direct SMS website messages from a mainland China website to establish accounts? Seems like every website uses SMS message verification instead of letting me sign up with an email.
When it comes to static HTML, the browser will just slowly grind along, showing the user what it is doing. It'll incrementally render the response as it comes in. Can't download CSS or images? No big deal, you can still read text. Timeouts? Not a thing.
Even if your Javascript framework is rendering HTML chunks on the server, it's still essentially hijacking the entire request. You'll have some button in your app, which fires off a request when clicked. But it's now up to the individual developer to properly implement things like progress bars/spinners, timouts, retries, and all the rest the browser normally handles for you.
They never get this right. Often you're stuck with an app which will give absolutely zero feedback on user action, only updating the UI when the response has been received. Request failed? Sorry, gotta F5 that app because you're now stuck in an invalid state!
This isn’t to exonerate SPAs, but I don’t think it helps to talk about it as a “JavaScript” problem because it’s really a user experience problem.
I have been using wasm and webgpu for visualization, partly to offload any burden from the embedded device to be monitored, but that could always be a third machine. Htmx says it supports websockets, is there a good way to have it eat a stream and plot data as telemetry, or is that time for a new tool?
I’m not necessarily disagreeing with your overall point, but modern JS is actually rather good at dealing with slow internet for server-client “applications”. It’s not necessarily easy to do, and there is almost no online resources that you can base your projects on if you’re a Google/GPT programmer. Part of this is because of the ocean of terrible JS resources online, but a big part of it is also that the organisations which work like this aren’t sharing. We have 0 public resources for the way we work as an example, because why would we hand that info to our competition?
I regularly talk to developers who aren't even aware that this is an option.
- https://web.whatsapp.com 11.12MB compressed / 26.17MB real.
- https://www.arstechnica.com 8.82MB compressed / 16.92MB real.
- httsp://www.reddit.com 2.33MB compressed / 5.22 MB real.
- https://www.trello.com (logged in) 2.50MB compressed / 10.40MB real.
- https://www.notion.so (logged out) 5.20MB compressed / 11.65MB real.
- https://www.notion.so (logged in) 19.21MB compressed / 34.97MB real. - Google Mail: Inbox is ~18MB (~6MB Compressed). Of that, 2.5MB is CSS (!) and the rest is mostly JS
- Google Calendar: 30% lower, but more or less the same proportions
- Confluence: Home is ~32MB (~5MB Comp.). There's easily 20MB of Javascript and at least 5MB of JSON.
- Jira: Home is ~35MB (~7MB compressed). I see more than 25MB of Javascript
- Google Cloud Console: 30MB (~7MB Comp.). I see at least 16MB of JS
- AWS Console: 18MB (~4MB Comp.). I think it's at least 12MB of JS
- New Relic: 14MB (~3MB Comp.). 11MB of JS.
This is funny because even being way more data heavy than the rest, its weight is way lower.
- Wiz: 23MB (~6MB Comp.) 10MB of JS and 10MB of CSS
- Slack: 60MB (~13MB Compressed). Of that, 48MB of JS. No joke.Javascript should even generally be fairly efficient in terms of bytes/capability. Run a basic minimizer on it and compress it and you should be looking at something approaching optimal for what is being done. For instance, a variable reference can amortize down to less than one byte, unlike compiled code where it ends up 8 bytes (64 bits) at the drop of a hat. Imagine how much assembler "a.b=c.d(e)" can compile into to, in what is likely represented in less compressed space than a single 64-bit integer in a compiled language.
Yet it still seems like we need 3 megabytes of minified, compressed Javascript on the modern web just to clear our throats. It's kind of bizarre, really.
Then of course every library is allowed to pin its own dependencies. So you can have 15 different versions of the same thing, so they can change API at will.
I poked around some electron applications.
I've found .h files from openssl, executables for other operating systems, megabytes of large image files that were for some example webpage, in the documentation of one project. They literally have no idea what's in there at all.
You talk about minification. The JS isn't minified much. Variable names are single letter, but property names and more aren't renamed, formatting isn't removed. I guess the minifier can't touch property names because it doesn't know what might get turned into JSON or not.
There's plenty of logging and span tracing strings as well. Lots of code like this:
_n.meta = {
name: "createThunk",
key: "createThunkaddEphemeralMessageSideEffectHandler",
description: "addEphemeralMessageSideEffect side effect handler"
};
Everything appears compiled using a coroutines framework, which adds some bloat. Not sure why they aren't using native async/await but maybe it's related to not being specialized based on execution environment.
Shooting from the hip, the learnings I'd take from this are:
1. There's a ton of low hanging fruit. A language toolchain that was more static and had more insight into what was being done where could minify much more aggressively.
2. Frameworks that could compile and optimize with way more server-side constants would strip away a lot of stuff.
3. Encoding logs/span labels as message numbers+interpolated strings would help a lot. Of course the code has to be debuggable but hopefully, not on every single user's computer.
4. Demand loading of features could surely be more aggressive.
But Slack is very popular and successful without all that, so they're probably right not to over-focus on this stuff. Especially for corporate users on corporate networks does anyone really care? Their competition is Teams after all.
For typedload I've had to use wget+sed to get rid of that crap after recompiling the website.
https://codeberg.org/ltworf/typedload/src/branch/master/Make...
Applies to all formal languages actually. 2^(8x20x10^6) ~= 2x10^48164799 is such a ridiculously large space...
A typical web page of text on a screen is about a kilobyte. Sure, you can pack more in with fine print, and obviously additional data is required to represent the styling, but the actual text is about 1 kb.
If you've sent 20 MB, then that is 20,000x more data than what was displayed on the screen.
Worse still, an uncompressed 4K still image is only 23.7 megabytes. At some point you might be better off doing "server side rendering" with a GPU instead of sending more JavaScript!
Some 7~10 years ago I remember I saw somewhere (maybe here on HN) a website which did exactly this: you gave it an URL - it downloaded a webpage with all its resources, rendered and screenshot'ed it (probably in headless Chrome or something), and compared size of png screenshot versus size of webpage with all its resources.
For many popular websites, png screenshot of a page indeed was several times less than webpage itself!
You can even swap in your own shared dictionary with a HTTP header, but trying to make your own dictionary specific to your content is a fool’s errand, you’ll never amortize the cost in total bits.
What you CAN do with shared dictionaries though, is delta updates.
https://developer.chrome.com/blog/shared-dictionary-compress... https://news.ycombinator.com/item?id=39615198
There are a few architectural/policy problems in web browsers that cause this kind of expansion:
1. Browsers can update large binaries asynchronously (=instant from the user's perspective) but this feature is only very recently available to web apps via obscure caching headers and most people don't know it exists yet/frameworks don't use it.
2. Large download sizes tend to come from frameworks that are featureful and thus widely used. Browsers could allow them to be cached but don't because they're over-aggressive at shutting down theoretical privacy problems, i.e. the browser is afraid that if one site learns you used another site that uses React, that's a privacy leak. A reasonable solution would be to let HTTP responses opt in to being put in the global cache rather than a partitioned cache, that way sites could share frameworks and they'd stay hot in the cache and not have to be downloaded. But browsers compete to satisfy a very noisy minority of people obsessed with "privacy" in the abstract, and don't want to do anything that could kick up a fuss. So every site gets a partitioned cache and things are slow.
3. Browsers often ignore trends in web development. React style vdom diffing could be offered by browsers themselves, where it'd be faster and shipped with browser updates, but it isn't so lots of websites ship it themselves over and over. I think the SCIter embedded browser actually does do this. CSS is a very inefficient way to represent styling logic which is why web devs write dialects like sass that are more compact, but browsers don't adopt it.
I think at some pretty foundational level the way this stuff works architecturally is wrong. The web needs a much more modular approach and most JS libraries should be handled more like libraries are in desktop apps. The browser is basically an OS already anyway.
I don't know exactly which features you are referring to, but you may have noticed that CSS has adopted native nesting, very similarly to Sass, but few sites actually use it. Functions and mixins are similar compactness/convenience topics being worked on by the CSSWG.
(Disclosure: I work on style in a browser team)
When it was decided that HTML shouldn't be versioned anymore it became impossible for anyone who isn't a full time and very conscientious web dev to keep up. Versions are a signal, they say "pay attention please, here is a nice blog post telling you the most important things you need to know". If once a year there was a new version of HTML I could take the time to spend thirty minutes reading what's new and feel like I'm at least aware of what I should learn next. But I'm not a full time web dev, the web platform changes constantly, sometimes changes appear and then get rolled back, and everyone has long since plastered over the core with transpilers and other layers anyway. Additionally there doesn't seem to be any concept of deprecating stuff, so it all just piles up like a mound of high school homework that never shrinks.
It's one of the reasons I've come to really dislike CSS and HTML in general (no offense to your work, it's not the browser implementations that are painful). Every time I try to work out how to get a particular effect it turns out that there's now five different alternatives, and because HTML isn't versioned and web pages / search results aren't strongly dated, it can be tough to even figure out what the modern way to do it is at all. Dev tools just make you even more confused because you start typing what you think you remember and now discover there are a dozen properties with very similar names, none of which seem to have any effect. Mistakes don't yield errors, it just silently does either nothing or the wrong thing. Everything turns into trial-and-error, plus fixing mobile always seems to break desktop or vice-versa for reasons that are hard to understand.
Oh and then there's magic like Tailwind. Gah.
I've been writing HTML since before CSS existed, but feel like CSS has become basically non-discoverable by this point. It's understandable why neither Jetpack Compose nor SwiftUI decided to adopt it, even whilst being heavily inspired by React. The CSS dialect in JavaFX I find much easier to understand than web CSS, partly because it's smaller and partly because it doesn't try to handle layout. The way it interacts with components is also more logical.
I'm not sure why Tailwind is magic. It's just a bunch of predefined classes at its core.
Let’s not kid ourselves, no matter how many fancy features, splitting, optimizing, whatever you do, JS webapps may be an upgrade for developers, they’re a huge downgrade for users in all aspects
It’s never faster than simply reloading the page. I don’t know what they were thinking, but they shouldn’t have.
On the other side, Gitlab doesn’t work with JS disabled.
I could put up with the glacial performance if it actually worked in the first place, but apparently adding whiz bang "AI" features is the only thing that matters these days.
The whole thing smacks of a rewrite so someone could get a bonus and/or promotion.
How my "colleagues" manage to get to 20 MB is a bit of mystery.
More often than not (and wittingly or not) it is effectively by using javascript to build a browser-inside-the-browser, Russian doll style, for the purposes of tracking users' behavior and undermining privacy.
Modern "javascript frameworks" do this all by default with just a few clicks.
One thing which surprised me at a recent job was that even what I consider to be a large bundle size (2MB) didn't have much of an effect on page load time. I was going to look into bundle splitting (because that included things like a charting library that was only used in a small subsection of the app). But in the end I didn't bother because I got page loads fast (~600ms) without it.
What did make a huge different was cutting down the number of HTTP requests that the app made on load (and making sure that they weren't serialised). Our app was originally going auth by communicating with Firebase Auth directly from the client, and that was terrible for performance because that request was quite slow (most of second!) and blocked everything else. I created an all-in-one auth endpoint that would check the user's auth and send back initial user and app configuration data in one ~50ms request and suddenly the app was fast.
Now take a good, hard look at the number of individual resources your application's page includes. Every tracker, analytics crapware, etc. gets in that queue. So do all the requests they generate. And the software you wrote is even slower to load because marketing insisted that they must have their packages loading at the top of the page.
Welcome to hell.
An extreme and outdated example, but back when cable modems first became available, online FPS players were astonished to see how much better the ping times were for many dial up players. If you were downloading a floppy disk of information, the cable modem user would obviously blow them away, but their round trip time sucked!
Like if you're on a totally reliable but low throughput LTE connection, the requests are NBD but the download is terrible. If you're on spotty 5g service, it's probably the opposite. If you're on, like, a heavily deprioritized MVNO with a slower device, they both super suck.
It's not like optimization is free though, which is why it's important to have a solid UX research phase to get data on who is going to use it, and what their use case is.
I agree a lot of full SPAs are poorly done, but some are do work well. FastMail is an example of a SPA done well.
The reason many SPAs are slower is just latency; traditional template-driven is:
- You send request
- Backend takes time to process that
- Sends all the data in one go.
- Browser renders it.
But full SPA is:
- You send request
- You get a stub template which loads JS.
- You load the JS.
- You parse the JS.
- You send some number of requests to get some JSON data, this can be anything from 1 to 10 depending on how it was written. Sometimes it's even serial (e.g. request 1 needs to complete, then uses some part of that to send request 2, and then that needs to finish).
- Your JS parses that and converts it to HTML.
- It injects that in your DOM.
- Browser renders it.
There are ways to make that faster, but many don't.
SPA’s also have a terrible habit of not behaving well after being left alone for a while. Nothing like coming back to a blank page and having it try to redownload the world to show you 3kb of text, because we stopped running the VM a week ago.
GitHub annoys the fuck out of me with this.
If I can retry a failed a network request that's fine. If I have to restart the entire flow when I get a failure that's unusable.
Imagine a Reddit or HN style UI with upvote and downvote buttons on each comment. If you have no JS, you have to reload the page every time one of the buttons is clicked. This takes a lot of time and a lot of packets.
If you have an offline-first SPA, you can queue the upvotes up and send them to the server when possible, with no impact on the UI. If you do this well, you can even make them survive prolonged internet dropouts (think being on a subway). Just save all incomplete voting actions to local storage, and then try re-submmitting them when you get internet access.
So no, "no JS at all" is not "by far the lightest weight" in many cases. This is just uncritically repeating dogma. Even 5K to 20K of JS can significantly increase performance.
However, you don't need to go full SPA. "No JS at all" and "SPA" are not the only options that exist. See my other comment: https://news.ycombinator.com/item?id=40541555
Sites like Hacker News, Stack Overflow, old.reddit.com, and many more greatly benefit from JS. I made GoatCounter tons faster with JS as well: rendering 8 charts on the server can be slow. It uses a "hybrid approach" where it renders only the first one on the server, sends the HTML, and then sends the rest later over a websocket. That gives the best of both: fast initial load without too much waiting, and most of the time you don't even notice the rest loads later.
Two of the lighter options right now though seem to be things like alpinejs, htmx, etc. Basic building blocks where / if needed.
I made a CLI for ultimateguitar (https://packages.debian.org/sid/ultimateultimateguitar) that works by grabbing the json :D
The only other alternative is to send the CSS inline, but that doesn’t work as well for caching for future page loads.
103 Early Hints is not nearly as useful as it doesn’t actually save a round trip. It only works around long request processing time on the server. Also, most web frameworks will have a very hard time supporting early hints, because it doesn’t fit in the normal request->response cycle, so I doubt it’s going get much adoption.
Also it would be nice to be able to somehow push 30x redirects to avoid more round trips.
Not even that, with outer space travel, we all need to build for very slow internet and long latency. Devs do need to time-travel back to 2005.
No need to invent time travel, just let them have a working retreat somewhere with only bad mobile connection for a few days.
I've seen some web sites with 250MB payloads on the home page due to ads and pre-loading videos.
I work with parolees who get free government cell phones and then burn through the 3GB/mo of data within three days. Then they can't apply for jobs, get bus times, rent a bike, top up their subway card, get directions.
Recently I've started to see some contracts with vastly more data, including some unlimiteds.
Even if I do web services I try to do so with as little moving parts as needed and with huge reliance on trusted and standardized solutions. If see any addition of complexity as a cost that needs to be weighed against the benefits you hope it brings you will automatically end up with a lean and fast application.
I always test my projects with throttled bandwidth, largely because (just like with a11y) following good practices results in better UX for all users, not just those with poor connectivity.
Edit: Another often missed opportunity is building SPAs as offline-first.
Yeah, there's a decent chance you'll be stuck with crappy internet as a result. I still remember quite a few of my employers getting frustrated that fibre internet wasn't available for the building they were renting office space in, despite them running a tech company that really needed a good internet connection.
You are going to get so many blank stares at many shops building web apps when suggesting things like this. This kind of consideration doesn't even enter into the minds of many developers in 2024. Few of the available resources in 2024 address it that well for developers coming up in the industry.
Back in the early-2000s, I recall these kinds of things being an active discussion point even with work placement students. Now that focus seems to have shifted to developer experience with less consideration on the user. Should developer experience ever weigh higher than user experience?
Developer experience is user experience. However, in a normative sense, I operate such that Developer suffering is preferable to user suffering to get any arbitrary task done.
Having lived in lots of countries (mainly developing) it’s embarrassing how bad our internet is in comparison
With fresh cache on "slow 3G", my site _works_, but has 5-8 second page loads. Would you have consider that usable/sufficient, or pretty awful?
(I work for a non-profit cultural heritage organization, we don't really have "conversions")
You can see these kinds of differences with just a few seconds difference, ideally I aim to stay under 2s, even on the slowest connection type. 2s is already very long for a user to wait and many will not.
Non profits are tricky. You could see volunteer sign ups and donations as conversions. I manage a non profit site as well and unfortunately I don't have a good solution that is both fast and approachable for our staff to use, so we had to make that compromise as well.
I take that time to clean a little, make a coffee, you know sometimes you gotta take a break and breathe. Life has gotten too fast and too busy and we all need a few reminders to slow down and enjoy the view. Thanks xfinity!
You want performance? Then include it in the requirements and give it the necessary time budget in a project.
Using Facebook level architectures for actually pretty basic needs can be like hitting an ant-sized problem with a sledgehammer and wondering why the sledgehammer is so heavy and awkward to swing for little things.
I feel you, but the experience can actually be very good if you invest a bit of time looking at Shadowsocks/V2Ray and building your own infra.
On the London Underground it's particularly noticeable that most apps are terrible at handling network that comes and goes every ~2 minutes (between stops), and which takes ~15s to connect to each AP as a train with 500 people on it all try to connect at the same time.
In Australia you're just 200ms from everything most of the time. That might not seem like much, but it really highlights which apps trip up on the N+1 request problem.
The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast.
And yeah, most of the internet works very badly when you drop the network all the time...
Summary: since 2024-05-06, users of all networks also get LTE in the U-Bahn thanks to a project between BVG and Teléfonica (not surprising that Teléfonica deployed the infra because they had the best U-Bahn LTE coverage beforehand)
Which provider are you with? Vodafone is still dead in large parts of the U-Bahn, but I know that one of them works much better.
I believe someone published a map of the data coverage of different providers on the berlin ubahn, but probably outdated now
I'm with sim.de which I believe is essentially an O2 reseller (apn references o2)
So… you can expect apps developed by engineers in London to get much worse on slow internet in 2025. :-)
Wifi inside the trains involves much more work, and to get them to ALSO be seamless across the entire setup - even harder. Easily 10x or 100x the cost.
It's sad, because the Internet shouldn't be that bad when the network drops all the time; it should just be slower as it waits to send good data.
> The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast.
The 200ms is telling.
I bet that WhatsApp is one of the rare services you use which actually deployed servers to Australia. To me, 200ms is a telltale sign of intercontinental traffic.
Most global companies deploy only to at most three regions:
* the US (us-east, us-central, us-east+us-east)
* Europe (west-europe),
* and somewhat rarely far-east (meither us-west or Japan)
This means that places such as south Africa, south America, and of course Australia typically have to pull data from one of these regions, which means latencies of at least 200ms due to physics.
Australia is particularly hit because, even with dedicated deployments in their theoretical catchment area, often these servers are actualy located in an entirely separate continent (west-us or Japan) and thus users do experience the performance impact of having packets cross half a globe.
So, I used to work at WhatsApp. And we got this kind of praise when we only had servers in Reston, Virginia (not at aws us-east1, but in the same neighborhood). Nowadays, Facebook is most likely terminating connections in Australia, but messaging most likely goes through another continent. Calling within Australia should stay local though (either p2p or through a nearby relay).
There's lots of things WhatsApp does to improve experience on low quality networks that other services don't (even when we worked in the same buildings and told them they should consider things!)
In no particular order:
0) offline first, phone is the source of truth, although there's multi-device now. You don't need to be online to read messages you have, or to write messages to be sent whenever you're online. Email used to work like this for everyone; and it was no big deal to grab mail once in a while, read it and reply, and then send in a batch. Online messaging is great, if you can, but for things like being on a commuter train where connectivity ebbs and flows, it's nice to pick up messages when you can.
a) hardcode fallback ips for when DNS doesn't work (not if)
b) setup "0rtt" fast resume, so you can start getting messages on the second round trip. This is part of noise pipes or whatever they're called, and tls 1.3
c) do reasonable-ish things to work with MTU. In the old days, FreeBSD reflected the client MSS back to it, which helps when there's a tunnel like PPPoE and it only modifies outgoing syns and not incoming syn+ack. Linux never did that, and afaik, FreeBSD took it out. Behind Facebook infrastructure, they just hardcode the mss for i think 1480 MTU (you can/should check with tcpdump). I did some limited testing, and really the best results come from monitoring for /24's with bad behavior (it's pretty easy, if you look for it --- never got any large packets and packet gaps are a multiple of MSS - space for tcp timestamps) and then sending back client - 20 to those; you could also just always send back client - 20. I think Android finally started doing pMTUD blackhole detection stuff a couple years back, Apple has been doing it really well for longer. Path MTU Discovery is still an issue, and anything you can do to make it happier is good.
d) connect in the background to exchange messages when possible. Don't post notifications unless the message content is on the device. Don't be one of those apps that can only load messsages from the network when the app is in the foreground, because the user might not have connectivity then
e) prioritize messages over telemetry. Don't measure everything, only measure things when you know what you'll do with the numbers. Everybody hates telemetry, but it can be super useful as a developer. But if you've got giant telemetry packs to upload, that's bad by itself, and if you do them before you get messages in and out, you're failing the user.
f) pay attention to how big things are on the wire. Not everything needs to get shrunk as much as possible, but login needs to be very tight, and message sending should be too. IMHO, http and json and xml are too bulky for those, but are ok for multimedia because the payload is big so framing doesn't matter as much, and they're ok for low volume services because they're low volume.
Do you know why there is a 4 device limit? I run into this limit quite a bit, because I have a lot more devices.
And... Why is there is WhatsApp for most commonly used devices, but iPads?
I was frustrated by this a while back, so I asked the PMs. Basically when investing engineering effort WhatsApp prioritises the overall number of users connected, and supporting iPads doesn't really move that metric, because (a) the vast majority of iPad owners also own a smartphone, and (b) iPads are pretty rare outside of wealthy western cities.
For iPad, I think it's like the sibling notes; expected use is very low, so it didm't justify the engineering cost while I was there. But I see some signs it might happen eventually [1]; WhatsApp for Android Tablets wasn't a thing when I was there either, but it is now.
For the four device limit, there's a few things going on IMHO. Synchronization is hard and the more devices are playing, the harder it is. Independent devices makes it easier in some ways because the user devices don't have to be online together to communicate (like when whatsapp web was essentially a remote control for your phone), but it does mean that all of your communications partner's devices have to work harder and the servers have to work harder, too.
Four deviced covers your phone, a desktop at home and work, and a laptop; but really most of the users only have a phone. Allowing more devices makes it more likely that you'll lose track of one or not use it for long enough that it's lost sync, etc.
WhatsApp has usually focused on product features that benefit the most users, and more than 4 devices isn't going to benefit many people, and 4 is plenty for internal use (phone, prod build, dev build, home computer). I'm sure they've got metrics of how many devices are used, and if there's a lot of 4 device users and enough requests, it's a #define somewhere.
[1] https://www.macworld.com/article/668638/how-to-get-whatsapp-...
This is a big one that makes low-bandwidth connections unusable in a lot of apps. The deluge of ad/tracking/telemetry SDKs' requests all being fired in parallel with the main business-logic requests makes them all saturate the slow pipe and usually leads to all of them timing out. By being third-party SDKs they may not even give you control of the underlying network requests nor the ability to buffer/delay/cache those requests.
One advantage of being Facebook in this case is that they're the masters of spyware and are unlikely to need to embed third-party spyware, so they can blend tracking/telemetry traffic within their business logic traffic and apply prioritization, including buffering any telemetry and sending it during less critical times.
When I was IT person on a research ship, WhatsApp was a nice easy one to get working with our "50+ people sharing two 256kbps uplinks" internet. Big part of that was being able to QoS prioritise the XMPP traffic which WhatsApp was a big part of.
Not having to come up with filters for HTTPS for IP ranges belonging to general-use CDNs that managed to hit the right blocks used by that app, was a definite boon. That, and the fact XMPP was nice and lightweight.
As far as I know google cloud messaging (GCN? GCM? firebase? Play notifications? Notifications by Google? Google Play Android Notifications Service?) also did/does use XMPP, so we often had the bizarre and infuriating very fast notifications _where sometimes the content was in the notification_ but when you clicked on it, other apps would fail to load it due to the congestion and latency and hardcoded timeouts TFA mentions.. argh.
But WhatsApp pretty much always worked, as long as the ship had an active WAN connection.... And that kept us all happy, because we could reach our families.
It's not exactly XMPP, it started with XMPP, but XML is big, so it's tokenized (some details are published in the European Market Access documentation), and there's no need for interop with standard XMPP clients, so login sequence is I think way different.
But it runs on port 5222? by default (with fallbacks to port 443 and 80).
I think GCM or whatever it's called today is plain XMPP (including, optionally, on the server to server side), and runs on ports 5228-5230. Not sure what protocol apple push is, but they use port 5223 which is affiliated with xmpp over tls.
So I think using a non 443 port was helpful for your QoS? But being avaialable on port 443 is helpful for getting through blanket firewall rules. AOL used to run AIM on all the ports, which is even better at getting through firewalls.
I once got asked "what was a life changing company/product" and my answer was WhatsApp - to slightly bemused looks.
WhatsApp connected the world for free. Obviously they weren't the first to try but when my (very globally distributed family) picked up WhatsApp in '09/'10 we knew we were onto something different. Being able to stay in touch with my brother half way across the world in realtime was very special. Nothing else at the time really competed. SMS was expensive and had latency. Email felt clunky and oddly formal - email clients don't feel "chatty". MSN was crap on mobile and you both had to be online. Ditto for Skype. For calls we even used to do this odd VOIP bridge where you would each call an endpoint for cheap international phone calls.
Meanwhile in 2012, I was able to install WhatsApp on my mum's old Nokia Symbian feature phone, use WhatsApp on a pay-as-you-go sim plan in Singapore communicating over WAP. The data consumption was so low I basically survived 2 months on maybe 1-2 top ups. Compare that with the other day where I turned on roaming on my phone (so I could connect to Singtel to BUY a roaming package) and my phone passively fetched ~50+ MB in seconds and I was hit with 400SGD of data charges (I was able to get them refunded)
I am very grateful to all the work and thought WhatsApp put into building an affordable global resilient communication network and I hope every one of the people involved got the payout they deserve.
Now (in 2024) have you tried Delta Chat?
I did once think whether a client could abstract over IMAP to build to build a WhatsApp-like UI/UX so clearly other people have thought the same.
Although I will confess I'm no longer really looking for a new chat experience...
How is the latency is for Delta Chat?
https://blog.cloudflare.com/bandwidth-costs-around-the-world
Most global companies (at least the US-based ones) have deployed in India, where I'm at right now. I suppose a billion people online is too big of a market to ignore (or not; I really don't know). Or there are services that I'm completely unaware of that's not in India.
Internet's pretty fast as well. Much faster than a certain conspicuous European country you'd expect to have fast internet ;)
There may be a setting to turn a browser back into a dumb wget visual displayer.
With wget it sends you the source you actually requested and doesn't try to get clever (stupid). Google likes WebP so that means everyone needs to join the WebP cargo cult even if it means transcoding a lossy format to another lossy format.
I was working from home (we had no offices yet) and my cable Internet got cut off. My only back up was a serial cable to a 2G Nokia 9000i. I had to re-encode a chunk of the music catalog at 8kbps so I could test it from home before I pushed the code to production.
Psychoacoustic compression is a miracle.
BTW, an interesting thing is that some/most carriers allow you to use CSD/HSCSD over 3G these days, and you can establish data CSD connection between two phone numbers, yielding essentially a dedicated L2 pipe which isn't routed over internet. Can have much lower latency and jitter if that's what you need. Some specialized telemetry is still using that, however as 3G is slowly getting phased out, it will probably have to change.
Still have an entire territory where everything is satellite fed (Nunavut), including its capital.
> Northwestel, one of biggest internet service providers in the North, said it provides broadband service for all Nunavut communities using Telesat's Telestar 19 VANTAGE high-throughput satellite. After the satellite was deployed in July 2018, Northwestel said it would significantly improve broadband connectivity in the territory, increasing speeds to 15 megabits per second.
https://www.ctvnews.ca/sci-tech/efforts-underway-to-improve-...
Interestingly, the same satellite (but different transponders) probably supplies internet to a good chunk of transatlantic voyages between US/Can and Europe: https://www.telesat.com/wp-content/uploads/2022/11/Telstar-1...
Websites that are Just Text On A Page should load fast, but many don't. Hacker News is blazing fast, but Google's API docs never load.
The worst problem is that most UIs fail to account for slow requests. Buttons feel broken. Things that really shouldn't need megabytes of data to load still take minutes to load or just fail. Google Maps' entire UI is broken.
I wish that developers spent more time designing and testing for slow internet. Instead we get data hungry websites that only work great on fast company laptops with fast internet.
---
On a related note, I run a website for a living, and moving to a static site generators was one of the best productivity moves I've made.
Instead of the latency of a CMS permeating everything I do, I edit text files at blazing speed, even when fully offline. I just push changes once I'm back online. It's a game changer.
Things aren't always that simple.
I'm in the UK, and my ping time to news.ycombinator.com is 147ms - presumably because it's not using a CDN and is hosted in the USA.
cloud.google.com on the other hand has an 8ms ping time.
So yes, Hacker News is a simple, low-JS page - but there can be other factors that make it feel slow for users in some places. This is despite me being in a privileged situation, having an XGS-PON fibre connection providing symmetric 8Gbps speeds.
At some point, I wrote a webapp (with one specific, limited function, of course) and optimized it to the point where loading it required one 27 kB request. And then turned up the cwnd somewhat, so that it could load in a single RTT :-) Doesn't really matter if you're in Australia then, really.
I have a domain that's currently marked as "dangerous" because I didn't use the latest version of Wordpress.
Now a days I download a 500mb google map cache on my phone and its like there is no point. Everything still has to fetch and pop in.
I used to work on the team that served those docs. Due to some unfortunate technical decisions made in the name of making docs dynamic/interactive they are almost entirely uncached. Basically every request you send hits an AppEngine app which runs Python code to send you back the HTML.
So even though it looks like it should be fast, it’s not.
Which I thought ties into the discussion about slow internet nicely.
It seems as though there should be accounting so you at least know what % of traffic went to you in the last hour (and a reference value of bandwidth_available divided by connected_users so you know what % was your share if everyone had equal need of it), if not a system that deprioritises everyone unless you punched the button that says "yes I'm aware what bandwidth I'm using in the next [X≤24] hour(s) and actually need it, thank you" which'll set the QoS priority for your MAC/IP address to normal
[1] Local-first software: You own your data, in spite of the cloud:
https://www.inkandswitch.com/local-first/
[2] Local-first Software:
[3] Leonard Kleinrock: Mr. Internet:
https://www.latimes.com/opinion/la-oe-morrison-use24-2009oct...
If you have access/the power (since these tend to be power hungry) to a general-purpose computing device and are willing to roll your own my suggestion is to use NNCP [1]. NNCP can can take data, chunk it, then send it. It also comes with a sync protocol that uses noise (though I can't remember if this enables 0RTT) over TCP (no TLS needed so only 1.5 RTT time spent establishing connection) and sends chunks, retrying failed chunks along the way.
NNCP supports feeding data as stdin to a remote program. I wrote a YouTube downloader, a Slack bot, a Telegram bot, and a Discord bot that reads incoming data and interacts with the appropriate services. On the local machine I have a local Matrix (Dendrite) server and bot running which sends data to the appropriate remote service via NNCP. You'll still want to hope (or experiment such) that MTU/MSS along your path is as low as possible to support frequent TCP level retries, but this setup has never really failed me wherever I go and let's me consume media and chat.
The most annoying thing on an international flight is that the NNCP endpoint isn't geographically distributed and depending on the route your packets end up taking to the endpoint, this could add a lot of latency and jitter. I try to locate my NNCP endpoint near my destination but based on the flight's WiFi the actual path may be terrible. NNCP now has Yggdrasil support which may ameliorate this (and help control MTU issues) but I've never tried Ygg under these conditions.
FWIW I think it's fine that major services do not work under these conditions, though I wish messaging apps did. Both WhatsApp and Telegram IME are well tuned for poor network conditions and do take a lot of these issues into account (a former WA engineer comments in this thread and you can see their attention to detail.) Complaining about these things a lot is sort of like eating out at restaurants and complaining at how much sodium and fat goes into the dishes: restaurants have to turn a profit and catering to niche dietary needs just isn't enough for them to survive. You can always cook at home and get the macros you want. But for you to "cook" your own software you need access to APIs and I'm glad Telegram, Slack, and Discord make this fairly easy. Youtube yt-dlp does the heavy lifting but I wish it were easier, at least for Premium subscribers, to access Youtube via API.
I find Slack to be the absolute worst offender networking-wise. I have no idea how, now that Slack is owned by Salesforce, the app experience can continue to be so crappy on network usage. It's obvious that management there does not prioritize the experience under non-ideal conditions in any way possible. Their app's usage of networks is almost shameful in how bad it is.
EDGE by itself isn't too bad on paper - you get a couple dozen kilobits per second. In reality, it was much worse. I ran into apps with short timeouts that would have worked just fine, if the authors had taken into account that loading can take minutes instead of milliseconds.
Low bandwith, high latency connections need to be part of the regular testing of software. For Linux, there's netem (https://wiki.linuxfoundation.org/networking/netem) that will let you do this.
An issue that the anonymous blog author didn't have was metered connections. Doing OS or even app upgrades was pretty much out of the question for cost reasons. Luckily, every few weeks or so, we got to a location with an unmetered connection to perform such things. But we got very familiar with the various operating systems' ways to mark connections as metered/unmetered disable all automatic updates and save precious bandwidth.
One size does not fit all. It would be a waste of time and effort to architect (or redesign) an app just because a residual subset of potential users might find themselves on a boat in the middle of the Pacific.
Let's keep things in perspective: some projects even skip testing WebApps on more than one browser because they deem that wasteful and an unjustified expense, even though it's trivial to include them on a test matrix, and this is a UI-only.
Why do you think this only applies to people on a boat?
Indeed, that's true. However, the number of users that go through similar experiences are quite low and even those who do are always a F5 away from circumventing that issue.
I repeat: even supporting a browser other than the latest N releases of Chrome is a hard sell to some companies. Typically the test matrix is limited to N versions of Chrome and the latest release of Safari when Apple products are supported. If budgets don't stretch even to cover the basics, of course that even rarer edge cases such as a user accessing a service through a crappy network will be far from the list of concerns.
And "local SIM cards" implies that you set foot on (is)lands to buy said SIM cards. Where did you only get 2G in the 2020s? I cannot believe any of this is still left in the South Pacific.
My previous smartphone supported 4G/3G/Edge, but for some reason the 4G didn't work. At all, ever, anywhere (not a provider/subscription or OS settings issue, and WiFi was fine).
In my country 3G was turned off a while ago to free up spectrum. So it fell back to Edge all the time.
That phone died recently. I'm temporarily using an older phone which also supports 4G/3G/Edge, and where the 4G bit works. Except... in many places where I hang out (rural / countryside) 4G coverage is spotty or non-existant. So it also falls back to Edge most of the time.
Just the other day (while on WiFi) I installed Dolphin as a lightweight browser alternative. Out in the countryside, it doesn't work ("no connection"), even though Firefox works fine there.
Apps won't download unless on WiFi. Not even if you're patient: downloads break somewhere, don't resume properly, or what's downloaded doesn't install because the download was corrupted. None of these issues over WiFi. Same with some websites: roundtrips take too long, server drops the connection, images don't load, etc etc.
Bottom line: app developers or online services don't (seem to) care about slow connections.
But here's the thing: for the average person in this world, fast mobile connections are still the exception, not the norm. Big city / developed country / 4G or 5G base stations 'everywhere' doesn't apply to a large % of the world's population (who do own smartphones these days, even if low-spec ones).
Not that some low-tier mobile plans also cap connection speeds. Read: slow connection even if there's 4G/5G coverage. There's a reason internet cafe's are still a thing around the world.
Re: Sunny - there's quite a bit of cloud cover and other devices onboard like the water maker and fridge (more important than Starlink!) also need a lot of power.
(The XS does have an Intel modem, known to be inferior to the Qualcomm flagship of the era)
I think there's just some of these areas where operational conditions make the towers break in some specific way.
I don't think that this is a valid assessment of the underlying problem.
Slow internet means many things, and one of them is connection problems. In connection-oriented protocols like TCP this means slowness induced by drop of packets, and in fire-and-forget protocols like UDP this means your messages don't get through. This means that slowness might take multiple forms, such as low data rates or moments of high throughput followed by momentary connection drops.
One solid approach to deal with slow networks is supporting offline mode, where all data pushes and pulls are designed as transactions that take place asynchronously, and data pushes are cached locally to be retried whenever possible. This brings additional requirements such as systems having to support versioning and conflict resolution.
Naturally, these requirements permeate onto additional UI requirements, such as support for manually synching/refreshing, displaying network status, toggling actions that are meaningless when the network is down, rely on eager loading to remain usable while offline, etc.
Inmarsat is an insecure, 2 Mbps (at best) connection with satellites at 22236 miles above Earth and a latency of about 900-1100 ms.
Starlink is a secure, 100 Mpbs (typical) connection with satellites at 342 miles above Earth and a latency of about 25 ms.
Odds of getting a video link on Inmarsat are low, and even if you do, it's potato quality. Source - have been using these systems operationally since the 1990s.
Polar coverage from GEO satellites is limited because how close to the horizon GEO satellites are from Pole. Pole uses old GEO satellites which are low on fuel and have relatively large inclinations... then you can talk to them for ~ 6 hours per 24.
Schedule: https://www.usap.gov/technology/1935/
Also the number of people worldwide whose only access to the internet is a $100 android phone with older software and limited CPU should be considered
As a luxury product used by a few people, maybe it "solves" the problem, but I don't think this is a very scalable solution.
The biggest issue for Starlink at the poles is, as you say, very sparse coverage. Also I suspect Starlink has to usually relay polar packets between satellites, not just a simple bent pipe relaying to a ground station.
The Braid Protocol allows multiple synchronization algorithms to interoperate over a common network protocol, which any synchronizer's network messages can be translated into.. The current Braid specification extends HTTP with two dimensions of synchronization:
Level 0: Today's HTTP
Level 1: Subscriptions with Push Updates
Level 2: P2P Consistency (Patches, Versions, Merges)
Even though today's synchronizers use different protocols, their network messages convey the same types of information: versions in time, locations in space, and patches to regions of space across spans of time. The composition of any set of patches forms a mathematical structure called a braid—the forks, mergers, and re-orderings of space over time.
Sometimes the bloat is unnoticeable on juicy machines and fast internet close to the DC. You can simulate that easily, but it requires the company to care. Generally, ad-tech and friends cares very little about small cohorts of users. In fact, the only reason they care about end users at all is because they generate revenue for their actual customers, ie the advertisers.
Sure, and it will keep being that way. But if this gets improved at the transport layer, seems like a win.
As an analogy, if buses are late because roads are bumpy and drivers are lousy, fixing the bumpy road may help, even if drivers don't change their behavior.
It really wouldn't. Lousy drivers are a way thinner bottleneck than the roads.
But it will improve the services where the drivers are good.
If the protocol is actually any good (its goals by themselves already make me suspicious it won't be), the well-designed web-apps out there can become even better designed. But it absolutely won't improve the situation people are complaining about.
What do you mean? TCP and HTTP is already designed for slow links with packet loss, it’s old reliable tech from before modern connectivity. You just have to not pull in thousands of modules in the npm dep tree and add 50 microservice bloatware, ads and client side “telemetry”. You set your cache-control headers and etags, and for large downloads you’ll want range requests. Perhaps some lightweight client side retry logic in case of PWAs. In extreme cases like Antarctica maybe you’d tune some tcp kernel params on the client to reduce RTTs under packet loss. There is nothing major missing from the standard decades old toolbox.
Of course it’s not optimal, the web isn’t perfect for offline hybrid apps. But for standard things like reading the news, sending email, chatting, you’ll be fine.
No. It will make things worse, because now lousy drivers are no longer constrained by the bumpy road, so they will become even lousier.
Case in point: software is shittier than ever, despite the 100x increase in computer performance since the 90s.
The optional Braid extension can _reduce_ complexity for offline-first apps, e.g. relative to WebDAV, https://news.ycombinator.com/item?id=40482610
You might be surprised at just how elegantly HTTP extends into a full-featured synchronization protocol. A key to this elegance is the Merge-Type: this is the abstraction that allows a single synchronization algorithm to merge across multiple data types.
As an application programmer, you will specify both the data types of your variables (e.g. int, string, bool) and also the merge-types (e.g. "this merges as a bank account balance, or a LWW unique ID, or a collaborative text field"). This is all the application programmer needs to specify. The rest of the synchronization algorithm gets automated by middleware libraries that the programmer can just use and rely upon, like his compiler, and web browser.
I'd encourage you to check out the Braid spec, and notice how much we can do with how little. This is because HTTP already has almost everything we need. Compare this with the WebDAV spec, for instance, which tries to define versioning on top of HTTP, and you'll see how monstrous the result becomes. Example here:
Braid is backwards-compatible with today's web, works in today's browsers, and is easy to add to existing web applications.. You can use Braid features in Chrome with the Braid-Chrome extension.
Here in the UK, some networks started shutting down 3G. Some have 2G as a low energy fall back, but we're supposed to use 4G/5G now. The problem is that 4G is not available everywhere yet, some areas until recently only had good 3G signal. So I've been dropping to 2G/EDGE more often than I'd like and a lot of stuff just stops working. A lot of apps are just not tested on slow, high latency, high package loss scenarios.
The problem with 3G as I understand it is that it uses more power and is less efficient than 4G/5G. They're starting to re-deploy the 3G bands as 4G/5G, so the other Gs will eventually benefit from this shutdown.
Try using Google Maps to get around on 2G :(
By the way, Google Maps supports offline maps. If the user can download the map (a few MB for a city), then at least the basic stuff will be fast.
- block windows update by returning DNS query for microsoft update endpoints as NXDOMAIN.
- use a captive portal to limit user session duration, so that unattended devices won't consume bandwidth.
- with freebsd dummynet, pfSense can share bandwidth equally among users. It can also share bandwidth by weight among groups. It helps.
- inside Arctic circle, the geosynchronous satellites are very low on the horizon and were blocked frequently when ship turns. I was able to read the ship's gyro and available satellites from VSAT controller and generate a plot to show the satellite blockage. It was so popular that everyone is using it to forecast next satellite online.
Things should be improving (sometimes fast, sometimes slowly) in that direction, but still is not something guaranteed everywhere, or at least in every place that your application is intended or needed to run. And there may be even setbacks in that road.
https://nncp.mirrors.quux.org/Use-cases.html
Also, relying on propietary OS'es is not recommended. Apple and iOS are disasters to work on remote, isolated places. No wonder no one uses Apple in Europe for any serious work except for iOS development. Most science and engineering setups will use anything else as a backend.
Most offline distros, like Ubuntu or Trisquel have methods of downloading the software packages for offline installs.
On chats, Slack and Zoom are disasters. Any SIP or Jabber client with VOIP support will be far more reliable as it can use several different protocols which can use far less bandwidth (OPUS for audio) without having to download tons of JS to use it, even if you cached your web app it will still download tons of crap in the background.
And, again, distros like Debian have a full offline DVD/BR pack which ironically can be better if you got that by mail. Or you can just use the downloaded/stored ISO files with
apt-cdrom add -m /path/to/your/file.iso
This way everything from Debian could be installed without even having an Internet connection.
I absolutely agree with the rest, though, including the part saying any "serious" software will have such features (and better support in general), and I second the examples you gave.
Until 2022 I had a rock-solid, never-failed 7 megabit/s-ish down, 640k up connection and I found it very easy to build sites that others describe as blazing fast.
This was slow really by the standards of much of the UK population even by 2015.
So all I had to do was make it fast for me.
A change of provider for practical reasons gave me an ADSL2+ connection that is ten times faster; still arguably slower than a lot of residential broadband in the UK but not so helpfully.
So now I test speed on mobile; even in the south east of England it is not that difficult to find poor mobile broadband. And when it’s poor, it’s poor in arguably more varied ways.
For example if you have a website that is meant to be used alongside a video call or while watching video, it’s difficult to really simulate all of that “feel”.
Using a link that is slow in practice is an invaluable experience.
In my experience, real life slow internet isn't like that. Packet loss numbers jump around, jitter switches second by second, speeds vary wildly and packets arrive out of order more than in order. Plus, with sattelites, the local router sends fake TCP acknowledgements to hide the slow data transfer, so the browser thinks it's connected while the traffic is still half a second away.
There are software tools to limit connectivity in a more realistic way, often using VMs, but they're not used as often as the nice browser speed limiter.
The other problem is that nobody in the software industry gives a damn. Everyone wants to make shiny apps with the last shiny tech. Try to mention optimizing for slow hardware/Internet and look at the face of your colleagues, behind their brand new M3.
I worked in a company with some remote colleagues in Africa. There were projects that they could literally not build, because it would require downloading tens of GB of docker crap multiple times a week for no apparent reason. The solution was to not have those colleagues work on those projects. Nobody even considered that maybe there was something to fix somewhere.
This is a good example of why I gave up a career as a JavaScript developer after 15 years. I got tired of fighting stupid, but even stupid woefully unqualified people need to make 6 figures spinning their wheels to justify their existence.
That's why I always cringe at these fancy-looking UI cross-platform apps since I know they will never work in a remote environment. Also, that is why offline support is very important. I only use Apple Notes and Things 3, both work tremendously in such remote settings.
Imagine your notes or to-do list (ehem Basecamp) not loading since it needs internet connection
Instead almost all apps are just a bad web browser that goes to one webpage.
With this technique, the server can break down a page load into smaller chunks of UI, and progressively stream smaller parts of the UI to the client as they become available. No more waiting for the entire page to load in, especially in poor network conditions as the author experienced from Anartica.
Not sure if this is serious or intended as a joke. It made me giggle nonetheless. Which is kind of sad.
Exaggeration for comedic effect, in other words.
Nice thought in theory but unnecessarily gives false hope.
Confluence/Jira sometimes need to download 20 megabytes in order to show a page with only text and some icons. I have a friend that tells me that in their company they had two jiras, one for developers, one for the rest of the company, because it was that dead slow.
I've lost all faith already that this will change for better.
No excuses! It is easier than ever to build fast, interactive websites, now that modern, native Javascript includes so many niceties.
Using JS dependencies is a minefield. At work, where I give less of a fuck, a dev recently brought in Material UI and Plotly. My god.
If you're targeting the general population (so, chat service, banking app, utility app,...), you should be targeting the majority of users, not just the new-flagship ones, so all the testing should be done on a cheapest smartphone you could buy in a supermarket two years ago (because well.. that's what "grandmas" use). Then downgrade the connection to 3g, or maybe even edge speeds (can be smulated on network devices), and the app/service should still work.
Somehow it seems that devs get the best new flagships only, optimize the software for that, and forget about the rest... and I understand that for a 3d shooter game or something, but an eg. banking app, should work on older devices too!
Blogs can be read from just about any device (or your favourite terminal). My blog, as an example: https://lmno.lol/alvaro
Shared more details at https://indieweb.social/@xenodium/112265481282475542
ps. If keen to join as an early adopter, email help at lmno.lol
Searching for an exact match of "PRL_ERR_WEB_PORTAL_UNEXPECTED" gives away Parallels as the first example of a hard-coded HTTPS timeout.
So on, so forth.
* Slack -- https://brr.fyi/media/engineering-for-slow-internet/load-err...
* Signal (main screen) -- https://brr.fyi/media/engineering-for-slow-internet/in-app-d...
* 1Password (config/about page) -- https://brr.fyi/media/engineering-for-slow-internet/in-app-d...
* Zoom (updater screen on Mac) -- https://brr.fyi/media/engineering-for-slow-internet/in-app-d...
* Parallels Desktop ("prl_client_app" is the binary name) -- https://brr.fyi/media/engineering-for-slow-internet/hardcode...
Anyone who has tried to use Slack:
- in the countryside with patchy connection - abroad - in China - on the London Underground
Can attest to how poor and buggy Slack is on bad internet.
These aren't weird edgecases - London is a major tech hub. Remote workers and open source communities rely on Slack around the world.
China is the second largest economy in the world with a population of 1.7B (incidentally it's blocked at least it was when I was last there but even on VPN it was weird and buggy).
How aren't these kinds of metrics tracked by their product teams. How isn't WhatsApp the gold standard now for message delivery, replicated everywhere.
Neither email nor WhatsApp have the weird consistency issues Slack has with simply sending a message with dodgy internet. Not to mention the unreliable and sometimes user-hostile client state management when Slack can't phone home which can sometimes lead to lost work or inability to see old messages you literally were able to see until you tried to interact with stuff.
Slack web downloads 40MB of Javascript. The macOS Slack client, that I guess should have all that stuff already, downloads 10MB of stuff just by starting it (and going directly to a private text only chat).
I work on the road (from a train / parking lot / etc) for five or six hours per week. My T-Mobile plan is grandfathered in, so I can't "upgrade" to a plan that allows full-speed tethering without considerably impacting my monthly bill.
Realistically, I hit around 1.5Mbps down. When Slack reloads itself, I have to stop _everything else_ that I'm doing, immediately, and give Slack full usage of my available bandwidth. Often times, it means taking my phone out of my pocket, and holding it up near the ceiling of the train, which (I've confirmed in Wireshark) reduces my packet loss. Even then, it takes two or three tries just to get Slack to load.
Apparently we must be very niche amongst their user base because these kinds of fixes hasn't made it onto their roadmap in years
To Amazon's credit, my ability to load the AWS console has improved considerably in recent months, but I can't say the same for Azure.
Infinitely resume-able, never looses progress, remains completely unfazed by timeouts, connection loss, etc. - and the ability to share received update-data between multiple devices peer-to-peer.
I went through such a process. Took me two weeks versus two hours using JS query
If by slider you mean carousel, HTML/CSS-only carousels lack a full range of interactivity without...scripting
And both those things don't really have anything to do with application frameworks.
I did find mention of increasing the cache¹ and lowering the refresh rate to 4 fps² (avoiding unnecessary animations), but I still feel the need for a server-side QUIC proxy that is less pushy based on network conditions. There is a red team project that has the protocol parsed out in Python³ instead of all the ActiveX control clients.
¹ https://superuser.com/questions/13487/how-to-increase-perfor...
² https://learn.microsoft.com/en-us/troubleshoot/windows-serve...
It enabled you to simulate network slow-downs, packet-loss, packet corruption, packet reordering and more. It was so critical in testing our highly network sensitive software.
In a place like that there are some sites that just don’t work sometimes because it is clear that the devs have never thought about what happens when a significant portion of your requests fail.
It's a great lesson in the importance of edge computing, but it also provides some harsh trusth about the current way we produce software. We cannot afford to deliver a spotty product. To get new updates out to all parties takes a prohibitively long time. This is hard for new people or outsiders giving courses to grok, and makes most modern devops practices useless to us.
The only way I've found to reliably check out a git repository over an unreliable link is to check it out somewhere reliable and `rsync` the .git directory over.
This does cause extra load on the servers, but if it's that big a problem for them, they can write the incremental patches themselves.
(I suspect that the "dumb http" transport are also incremental if you squint hard enough at them, but I've never had reason to investigate that closely)
In the dial-up era things used to be unworldly fast merely by getting access to something like 10M ethernet. Now mobile connections are way, way faster than physical connections in the 90's but web pages aren't few KB, they are few MB at minimum.
It takes four seconds and 2.5 MB to load my local meteorological institute's weather page which changes not more often than maybe once an hour and could be cached and served as a static base page in a few dozen milliseconds (i.e. instantly). A modern connection that's plenty capable to support all my remote work and development over a VPN and interactive shells without any lag can't help me get modern web pages load any faster because of the amount of data and the required processing/execution of a million lines of javascript that's imported from a bunch of number of sources, with the appropriate handshake delays of new connections implied, for each page load.
A weather page from 2004 served exactly the same information as a weather page from 2024, and that information is everything required to get a sufficient glimpse of today's weather. One web page could be fixed but there are billions of URIs that load poorly. The overall user experience hasn't improved much, if at all. Yes, you can stream 4K video without any problems which reveals how fast things actually are today but you won't see it when browsing common pages -- I'd actually like to say web pages have only gone slower despite the improvements in bandwidth and processing power.
When many pages still had mobile versions it was occasionally a very welcome alternative. Either the mobile version was so crappy you wanted to use the desktop version on your phone, or it was so good you wanted to predominantly load the mobile version even on desktop.
I'd love to see an information internet where things like weather data, news articles, forum posts, etc. would be downloadable as snippets of plaintext, presumably intended to be machine readable, and "web" would actually be a www site that builds a presentation and UI for loading and viewing these snippets. You could use whichever "web" you want but you would still ultimately see the same information. This would disconnect information sources from the presentation which I think is the reason web sites started considering "browser" a programmable platform, thus taking away user control and each site bloating their pages each individually, leaving no choice for the user but maybe some 3rd party monkeyscripts or forced CSS rules.
If the end user could always choose the presentation, the user would be greatly empowered in comparison to the current state of affairs where web users are currently being tamed down to be mere receivers or consumers of information, much not unlike passive TV viewers.
One key difference is that client software was installed locally, which (in modern terms) decouples UI from content. As the article points out, the actual data you're dealing with is often measured in bytes. An email reader or AOL Instant Messenger would only have to deal with that data (plus headers and basic login) instead of having to download an entire web app. And since the protocols didn't change often, there was no need to update software every few weeks (or months, or even years).
Another key difference, which is less relevant today, is that more data came from servers on the local network. Email and Usenet were both designed to do bulk data transfer between servers and then let users download their individual data off of their local server. As I recall, email servers can spend several days making delivery attempts before giving up.
I’m currently on a moving cruise ship in the Mediterranean with a starlink connection. A latency of 300-500 ms seems to be normal. Although bandwidth is tolerable at 2-4 mbps during the day with hundreds of passengers using it. At night it gets better. But latency can still be frustrating.
Despite its usage issues, its built with reliability in mind and its a great way to transfer large files.
The author clearly says that downloaders elaborate enough to deal with slow connections (e.g. "download in a browser") were fine. The problem is that modern apps don't let you download the file the way you want, they just expect you to have a fast internet connection.
Bittorent has webseeds support, which can use apples direct CDN urls to create a torrent file to download. archive.org still uses this technique and AWS S3 used to do this when they had torrent support.
There a website that do just that, it creates a torrent file from any direct weburl.
Do I understand correctly that one can make a torrent from any HTTP-available file (without even owning the file) and start downloading and sharing with BitTorrent? It's incredible!
Unless you're specifically talking about bases in Antarctica?
No different from regional Australia, which ALWAYS has some crippling addiction, whether it's alcohol, domestic violence or something else. Not the best environment.
Not to mention having a good UX and explaining to the customer what's going on, helping with session resumption. I regret it. Couldn't make it through :(
This problem will return with a vengeance once humans occupy the Moon and Mars.
PSA: Please optimize your website, web apps for caching and efficiency, and offer slow/graceful fallback versions instead of 8K fullscreen video as your homepage for all users.
My favorite setup when working remotely with slow internet is the following :
- Build tasks & large data operations run on my home server via SSH
- Markdown notes, Notion being unusable
- 100% CLI apps, which are usually the lightest and most efficient
Everything is split into chunks. Downloads and uploads happen whenever connections can happen. If someone local has a copy, they can seed it to you without you needing an external connection. You can have a cache server that downloads important stuff for everyone.
Most of our web libraries and frameworks are indeed quite bloated (with features of convenience), downloading 20 MB of JS and 50 MB of content in total to render a page is insane when you think about it. We'd need to be able to turn off most images or visual elements to focus purely on the elements and their functionality, except for cases where displaying an image is critical for the function (and even then give the choice of showing a low quality version with a smaller file size). Things like web safe fonts that are already present in the browser/OS, most likely no libraries like React or Vue either (maybe Preact or Svelte).
We'd need to allow for really long request (say, fetch) timeout values, maybe even to choose how to set them based on the connection quality (if a user has a really fast connection but a request suddenly hangs and is taking upwards of a minute, something has probably gone wrong and it'd make sense to fail that request, vs a user in a remote area for whom all requests are similarly slow), assuming that the server doesn't mind connections that linger around for a long time at slow speeds.
We'd also need to allow for configuring an arbitrary HTTP cache/proxy for any site visited and file requested (say, store up to 1 TB on some local server based on file hashes and return the same for any user that requests that), but obviously things don't usually work that way over privacy/security concerns (nowadays different sites even download duplicate copies of the same files due to changes in the browsers: https://www.peakhour.io/blog/cache-partitioning-firefox-chro... ). Maybe even for any web request that the OS might want to do, like system updates, basically a full on MitM for the whole system.
Speaking of which, no more Electron or large software packages. Only native software with like Win32/WPF or maybe something like GTK/Qt, but nowadays it seems like even phone apps, not just desktop software, often don't use the system GUI frameworks, but instead ship a bunch of visual fluff, which might look nice and work well, but also takes up a bunch of space.
I don't think there are incentives out there to guide us towards a world like that, which doesn't quite make sense to me. Lightweight websites should lead to better customer/user retention, but in practice that doesn't seem like something that anyone is optimizing for - ads everywhere, numerous tracking scripts, even autoplay videos, for everything from news sites to e-commerce shops.
People who do optimize for that sort of stuff, seem to be a part of a smaller niche enthusiast community (which is still nice to see), like:
Admittedly, even I'm guilty of bloating my homepage size from ~150 KB to ~ 600 KB due to wanting to use a custom set of fonts (that I host myself), even WOFF2 didn't save me there.
> If you have the ability to measure whether bytes are flowing, and they are, leave them alone, no matter how slow. Perhaps show some UI indicating what is happening.
Allowing this means easy DDOS attack. An attacker can simply keep thousand of connections open
I personally think all apps benefit from being less reliant on a stable internet connection and that's why there's a growing local-first movement and why I'm working on Triplit[1].
[1] www.triplit.dev
hyperwave is great for slow connections - in my testing, even a 2G throttled connection is still usable with load times in the 5s range.
Technology today is developed by, and for, privileged people. Has been that way for a while. Ever since you had to upgrade your computer in order to read the news, there has been a slow, steady slog of increasing resource use and conspicuous consumption.
I remember using the 9600 baud modem to get online and do the most basic network transactions. It felt blazing fast, because it was just some lines of text being sent. I remember the 2.5KBps modem, allowing me to stream pictures and text in a new World Wide Web. I remember the 5KBps modem making it possible to download an entire movie! (It took 4 days, and you had to find special software to multiplex and resume cancelled downloads, because a fax on the dialup line killed the connection) I remember movies growing to the size of CDROMs, and later DVDROMs, so those who could afford these newer devices could fit the newer movies, and those who couldn't afford them, didn't. I remember the insane jump from 5KBps to 1.5Mbps, when the future arrived. Spending days torrenting hundreds of songs to impress the cool kids at school, burning them CDs, movies, compiling whole libraries of media [hey, 15 year olds can't afford retail prices!].
I remember when my poor friends couldn't use the brand new ride-sharing services Uber and Lyft because you had to have an expensive new smartphone to hail them. They'd instead have to call and then pay for a full fare taxi, assuming one would stop for them in the poor neighborhood, or wait an hour and a half to catch the bus. I remember when I had to finally ditch my gaming laptop, with the world-class video card you could've done crypto mining on, because opening more than 5 browser tabs would churn the CPU and hard-drive, max out the RAM, and crash the browser. I remember having to upgrade my operating system, because it could no longer run a new enough browser, that was now required to load most web pages. I remember buying smartphone after smartphone after smartphone - not because the previous one stopped working, but because more apps required more cpu and more memory and more storage. I remember trying to download and run a chat app on my local machine, and running out of memory, because the chat app had an embedded web browser. I remember running out of my data cap on my cell phone because some app decided it wanted to stream a load of data as if it was just unlimited. I remember running out of space on my smartphone because 70% of the space was being used just to store the Operating System files.
I'm not complaining, though. It's just how the world works. Humanity grows and consumes ever more resources. The people at the top demand a newer, better cake, and they get it; everyone else picks up the crumbs, until they too get something resembling cake. I sure ate my share. Lately I try to eat as little cake as possible. Doesn't change the world, but does make me feel better. Almost like the cake is a lie.
So Google Fiber needed a pure JS Speedtest for installers to verify connections. Installers were issued with Chromebooks, which don't support Flash and the Ookla Speedtest at the time used Flash. There's actually good reasons for this.
It turns out figuring out the maximum capacity of a network link is a nontrivial problem. You can crash the browser with too much traffic (or just slow down your reported result). You can easily under-report speed by not sending enough traffic. You have to weigh packet sizes with throughput. You need to stop browsers trying to be helpful by caching things (by ignoring caching headers). There's a long list.
So I did get a pure JS Speedtest that could actually run up to about ~8.5Gbps on a 10GbE link to a Macbook (external 10GbE controller over TB3).
You learn just how super-sensitive throughput is to latency due to TCP throttling. This is a known and longstanding problem, which is why Google invested in newer congestion control schemes like BRR [1]. Anyway, adding 100ms of latency to a 1GbE connection would drop the measured throughput from ~920-930Mbps to a fraction of that. It's been a few years so I don't remember the exact numbers but even with adjustments I recall the drop off being like 50-90%.
The author here talks about satellite Internet to Antarctica that isn't always available. That is indeed a cool application but you don't need to go this extreme. You have this throughput problem even in Australia because pure distance pretty much gives you 100ms minimum latency in some parts and there's literaly nothing you can do about it.
It's actually amazing how much breaks or just plain sucks on that kind of latency. Networked applications are clearly not designed for this and have never been tested on it. This is a general problem with apps: some have never been tested in non-perfect Internet conditions. Just th eother day I was using one of the Citi-bike apps and it could hang trying to do some TCP query and you'd every now and again get "Connection timed out" pop ups to the user.
That should never happen. This is the lazy dev's way of just giving up, of catching an exception and fatalling. I wish more people would actually test their experience when there was 100ms latency or if there was just random 2% packet loss. Standard TCP congestion control simply doesn't handle packet loss in a way that's desirable or appropriate to modern network conditions.
[1]: https://cloud.google.com/blog/products/networking/tcp-bbr-co...
How can I give priority to one of these instances?
But to add something useful as well, you could just ctrl+z the ones you want to pause to let the priority one finish?
Take Home Depot for example. Loading their website in a mobile browser is soooooooooo slow. The rendering is atrocious, with elements jumping all over the place. You click on one thing and it ends up activating a completely different element, then you have to wait for whatever you just clicked to load and jump all over the place again. Very frustrating! Inside their stores is even worse! I asked for help locating an item from one of their workers one day and they pull up their in-store app. That too was slower than molasses and janky, so we ended up standing there for several minutes just chatting waiting for it to load.
Another thing I think people should think about is scenarios with intermittent connectivity where there is literally no internet for periods ranging from minutes to days.
Sadly in both these regards I believe we're utterly screwed.
Even the Offline First and Local First movements who you'd think would handle these issues in at least a semi-intelligent manner don't actually practice what they preach.
Look at Automerge or frankly the vast majority of the other projects that came out of those movements. Logically you'd think they have offline documentation that allows people to study them in a Local First fashion. Sadly that's not the case. The hypocrisy is truly a marvel to behold. You'd think that if they can get hard stuff like CRDTs right they'd get simple stuff right like actually providing offline / local first docs in a trivial to obtain way. Again sadly not.
The following two links are yet another example of a similar kind of hypocrisy: https://twitter.com/mitchellh/status/1781840288300097896 https://github.com/hashicorp/vagrant/issues/1052#issuecommen...
Again at this point the jokes are frankly writing themselves. Like bro make it possible for people to follow your advice.
Also if you directly state or indirectly insinuate that your tool is ANY/ALL OF Local First, or Open Source, or Free As In Freedom you better have offline docs.
If you don't have offline docs your users and collaborators don't have Freedom 1. If you can't exercise Freedom 1 you are severely hampered in your ability to exercise Freedoms 0, 2, or 3 for any nontrivial FOSS system.
The problem has gotten so bad the I started the Freedom Respecting Technology movement which I'm gonna plug here: https://makesourcenotcode.github.io/freedom_respecting_techn...
The purpose of broadband is so that nothing has any latency when you're not on dial-up.
Except user-initiated mega downloads, and they occur at the maximum connection rate the hardware can handle. At bandwidth costs anybody can afford. Just like with dial-up, why settle for less instantaneous performance than the hardware can handle? Just to make it so lesser geeks can participate?
The hallmark of insufficient web "engineering" has always been the completeness of the failure to pre-deploy on dial-up.
This is probably about as close to experiencing an Antarctic data satellite connection within the continental US as is possible, and would be even worse than a 56k modem; While the bandwidth is roughly the same, Red Pocket users often report latencies in excess of 1000+ms, whereas 56k modem latency was typically under 200ms. The frequent disconnects (because we're far from the tower) would simulate Antarctic conditions, as well.
So how do internet-connected people living in 2024 survive such an atrocity?
First, we would leverage SSH and CLI tools as much as possible. Brow.sh and Carbonyl browser for web and Mutt for email. Mosh.sh to keep the SSH connection alive and reconnect automatically on reconnect--even if your IP changes. (Mosh is incredible.) Some terminals have a zoom feature (ctrl++ or - in kitty) which may help if some web browser item is just too difficult to see in its blocky resolution. See also the Carbonyl --bitmap and --zoom flags: https://github.com/fathyb/carbonyl/releases/tag/v0.0.3
The youtube-dl command can fetch tiny versions of a video on a server (even audio-only if we don't care about the video portion such as with long-form interviews), then transcode them on the server to the smallest-tolerable bitrate. rsync them home overnight and watch them the next day. rsync can retry on failure: rsync --partial --progress --compress-choice=STR, --zc=zlib --compress-level=9 --rsh=ssh user@host:remote_file local_file
Edit: On second thought, rsync doesn't perform well in high-latency environments due to TCP acks, but something like uftp which uses UDP should do better, and offers resume. https://stackoverflow.com/questions/38014501/what-is-the-fas... https://uftp-multicast.sourceforge.net/ https://sourceforge.net/p/uftp-multicast/discussion/general/...
Needs a remote Linux server, but ServerHunter has servers adequate for the task for less than $3/mo. Focus on RAM; Most any plan has sufficient hard drive space and speed, sufficient bandwidth and CPU for what you're doing. You're not serving large content to many demanding customers, and in my personal testing I have found that even 1GB RAM should be enough--but obviously more is better. And many companies only charge for data egress--which should be smaller than ingress. Data ingress to the server is usually free, and when you're downloading large amounts of data to the server, then compressing them, egress is much smaller. Choose companies that have a domain age of at least 1 year old; I've had a fly-by-night company take my money and run. (PayPal reimbursed me.)
To download large files, try first compressing them; Although, depending on the original file, compression can either give you no advantage or even _increase_ the file size. Try different compression methods; bzip2, zpaq, xz, lzma, etc. ls -l the before and after to see if it helped any.
Also, we'd rely on phones apps for things like Gmail/Facebook/Messenger/WhatsApp/Telegram/Twitter/Discord/etc. This saves always redownloading JavaScript for each visit. While there are Windows/Linux desktop clients for these, they usually are only thin layer over a browser, which means they still download JavaScript and large images on every use. But a phone app has everything it needs all the time--until the next app update, of course. Can temporarily disable app updates, but I wouldn't do that for long. The good part about app updates is they can tolerate slow links and frequent disconnects. And we often take our phones with us to high-bandwidth locations, so we can do app updates there and disable them over cell data. The article said on some days Antarctica has better bandwidth availability, so you could halt updates until conditions improve.
Same with Windows or Linux OS auto-updates; Disable them until bandwidth conditions improve. (But you're probably already doing that.) Edit: After reading the article again, yes they definitely are doing that, and are still having problems. Bummer.
Installing Bluestacks is an option, for the above reason. Using phone apps on the desktop that don't need to download JavaScript on every use could bridge the gap somewhat. Disable its auto-updates as well.
Gmail can be set up for IMAP or POP3. Try both; POP should give a better experience, but by default it removes messages on the server, so look for that setting to disable it if you want to keep them there. I believe you can disable downloading attachments and images automatically, which probably works better for IMAP? Not sure if POP can handle this. Doing it this way means you don't need to load the web interface any time you check mail, and you'll only be reading the text content of emails. It should therefore work through Mutt as well.
Then there's browser tricks for local rendering: Disable graphic images and disable media autoplay. uBlock Origin to reduce overall download size. Turn off JavaScript until truly needed. Pain in the butt, but it works.
In Windows, set your network connection to metered. This should help. (And back in the US, we can set our phones to low bandwidth mode--but that would only help users on cell networks; I'm sure the south pole lacks cell reception.)
You're not going to be gaming with large downloads and patches, nor will you be competing with low-latency gamers around the globe. But not all games need these. Play against the computer instead of other people. Play only pre-downloaded or DVD-based games. Or go back to the 90s and set up a LAN party and play against one another in the same room. We love Unreal Tournament 1999 GOTY; Plays blisteringly fast on any computer, is cheap to buy on DVD or Steam, has no in-game paid-for upgrades, has oodles and oodles of maps for download, and gameplay still feels fresh even 25 years later. Card games and offline phone games are king in low bandwidth environments.
You're probably not going to make many audio or video calls, so learn to send mp4s or mp3s to others back and forth by email. Use the lowest-tolerable bitrate. A 6Kbps OPUS/32Kbps AV1 video is about 5KB/sec and is clear enough to follow along, and would certainly be acceptable for a video message. A 0.5Kbps OPUS-encrypted audio file is surprisingly understandable, if not very enjoyable. But to get the message across, it works. A 6Kbps OPUS audio sounds like telephone quality to my ears. https://heavydeck.net/post/64k-is-enough-for-video/
Remember that 24-64kbit audio streams are still a thing. Search low bandwidth streaming radio. Sort by bitrate: https://directory.shoutcast.com/ https://directory.shoutcast.com/?q=299_Talk_297
I don't know if podcast apps have a low-bandwidth option, but it's worth investigating. Apparently the Player FM Pro app compresses pods on their servers before sending.
There are a few text-only news sites: https://blog.wturrell.co.uk/text-only-news-websites/
RSS feeds can fetch news over a slow link, and they're still a thing: https://rss.feedspot.com/world_news_rss_feeds/
Finally, there's the possibility of running a remote desktop session: Something like RDP over SSH, VNC over SSH, NoMachine's NX, TeamViewer, etc. Can give clearer detail than brow.sh and can sometimes be faster than browsing locally. I tested it by intentionally limiting my desktop's speed to 5KB/sec and while it wasn't fun, it got the job done. Took about ten seconds to paint a page on a 1GB VM, but that can often be faster than rendering a JavaScript-heavy page locally. Seems to be a good compromise for certain situations when the rest of the options fail you.
Typing is too laggy (5 seconds before characters appeared) but one way around that is to type into a file via SSH then cat the file on the server, paste that into the web form. Or write your notes to a file on your local system and copy+paste that directly in. I've found that even a server with 1GB RAM is sufficient for even this task, as long as you're only looking at one web page at a time. No multiple tabs. I ran XFCE4 on Ubuntu 22.04 and Google's official Chrome package. Resized the connection to 1024x768, which I consider to be the minimum for useful browsing. Changed the quality settings to lowest, color depth to lowest, etc. Disabled audio and printing.
To read large pages offline, print to PDF on the server inside RDP/VNC and download that with rsync.
The combination of all of these should help us get through a season of needing to save money, and it may very well save the day in Antarctica, as well.
People should not be limited from employing already existing infrastructure to overcome their edge-case troubles just because that infrastructure is not exposed due to it being unnecessary to the "majority of the clients".
There are lots of places in the world where the internet speed isn't great.
I'm actually experiencing slow internet right now and I' m in Germany.
for example, the audience of a site might be in southern Africa, with same bad connectivity, but the EU/UN site developers are in the north so they don't care and the consequences are in program adoption that fails to blame that. Or you might be doing business with a coffee producer and now your spiffy ERP is missing data because it's too much effort for them to update the orders, so your procurement team have to hire an intern to do it over the phone costing you an extra 2k a month. Or which is more likely for the crowd here, yall losing clients left and right because your lazy sysadmin blocked entire country ip ranges because once they saw a single DoS wave from there.