If there is a slight positive note to all this, then it is that these outages are so large that customers usually seem to be quite understanding.
I have always felt so, but my opinion is definitely in the minority.
In fact, I find that folks have extremely negative responses to any discussion of improving software Quality.
A large proportion of “developers” enjoy build vs buy arguments far too much.
Now that we have an abundance of compute and most people run devices more powerful than the devices that put man on the moon, it's easier than ever to make app bloat, especially when using a framework like Electron or React Native.
People take it personally when you say they write poor quality software, but it's not a personal attack, it's an observation of modern software practices.
And I'm guilty of this, mainly because I work for companies that prioritize speed of development over quality of software, and I suspect most developers are in this trap.
The typical argument that I see, is homemade encryption, which is quite valid.
However, encryption is just a tiny corner of the surface.
Most folks don’t want to haul in 1MB of junk, just so they can animate a transition.
Well, I guess I should qualify that: Most normal folks wouldn't want to do that, but, apparently, it's de rigueur for today's coders.
Not really sure how our community is supposed to deal with this.
Which only shows that chasing five 9s is worthless for almost all web products. The idea is that by relying on AWS or Cloudflare you can push your uptime numbers up to that standard, but these companies themselves are having such frequent outages that customers themselves don't expect that kind reliability from web products.
It's monthly by now
If I choose something else, we're down, and our competitors aren't, then my overlords will start asking a lot of questions.
I think at the very least, one should plan the ability to switch to an alternative when your main choice fails… which together with AWS and GitHub is a weekly event now.
(Note: Zero negative sentiment towards imgur here)
Most time I get ddosed now it's either Facebook directly, Something something Azure or any random AI.
And lots of real users time wasted for captchas.
Does it make sense? Nah, but is it part of the weird reality we live in. Looks like it
I have no way of contacting Facebook. All I can do is keep complaining on hackernews whenever the topic arrises.
Edit:// Oh and I see the same with Azure, however there I have no list of IPs to verify it's official just because it looks like it.
Tbh though this is sort of all the other companies fault, "everyone" uses aws and cf and so others follow. now not only are all your chicks in one basket, so is everyone elses. When the basket inevitably falls into a lake....
Providers need to be more aware of their global impact in outages, and customers need to be more diverse in their spread.
So you think the problem is they aren't "aware"?
Outages happen, code changes occur; but you can do a lot to prevent these things on a large scale, and they simply dont.
Where is the A/B deployment, preventing a full outage? What about internally, where was the validation before the change, was the testing run against a prodlike environment or something that once resembled prod but hasnt forever?
They could absolutely mitigate impacting the entire global infra in multiple ways, and havent, despite their many outages.
> this is tenable as long as these services are reliable
do you hear yourself, this is supposed to be a distributed CDN. imagine if HTTP had 30 minutes of downtime a year.
and judging by the HN post age, we're now past minute 60 of this incident.
Huh? It's been back up during most of this time. It was up and then briefly went back down again but it's been up for a while now. Total downtime was closer to 30 minutes
CF can be just as difficult if not more to migrate off of especially when using things like durable objects