Hacker News Reader

The "experts" in this case are

> Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19

Dr. Cath-Speth has a PhD in cultural anthropology

> Cori Crider, the executive director of the Future of Technology Institute

A lawyer

> Madeline Carr, professor of global politics and cybersecurity at University College London

A professor. Her bio doesn't say what her degree is in, but she mostly seems to publish in political science and international relations

So, not a single technical expert. Not anyone who has ever run a hosting service before or even worked for one. Just people who write papers and sit around waiting for journalists to call them for quotes.

Opinions are valid but also worthless. Just give me a funny tweet to digest the situation.

Do you not think it a bit too hyperbolic to throw scare quotes around experts and imply the only people who can have opinions on systemic risk are software engineers? I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers.

Anyone can have an opinion, I never said or implied otherwise. Having an opinion does not make one an expert, hence the scare quotes.

The headline is misleading because when there is news about experts saying something about technology, one would naturally think that they are at least somewhat technical experts. Instead the "expert" is the director of the "Big Tech is Bad Institute" who says that "Big Tech is Bad". And their qualification of being an expert is solely that they are director of the "Big Tech is Bad Institute".

And one would hope that the stats being quoted about desktop share were from someone who has been at that research firm in the last 20 years or so. I'm not sure how active he is at all at this point. I have a feeling someone looking for some stats found something old that may or may not have actually had a date on it.

(If I'm wrong mea culpa but I'm pretty sure.)

> when there is news about experts saying something about technology, one would naturally think that they are at least somewhat technical experts.

But the experts here are not "saying something about technology". Rather they are saying something about uses of technology. So they don't need to be cloud engineers or know anything about datacenters, at all, really. What would be required (and here you may have a leg to stand on) is expertise in social and economic aspects of (now) critical infrastructure.

I don’t think it’s useful at all.

What are they going to say that’s useful for making concrete technical decisions?

They can advise on how to write contracts for dealing with these situations after the fact, I suppose.

> I don't think it is unreasonable for people who haven't run or worked for a hosting service to have opinions on the policy aspect or economic impact of hyperscalers.

Yeah, that's completely fair. My angle was more that firstly this doesn't come across as an opinion that needs the expert in question, and secondly this is yet another case of 'Talk is cheap, show me the code', particularly when quotes in the article include "We urgently need diversification in cloud computing."

I feel like the 'We' is doing an awful lot of heavy lifting and there's no mention of the costs of taking on such a task.

Additionally, and awkwardly, it's possible to be both a monopoly in the space but also technically a more stable solution, making the cost for competitors or people willing to use competitors doubly high.

Edit: Realised afer the fact I'm GP to your post, assumed it was mine, keeping the words anyway.

I don't think anyone needs to produce any code. I've worked at companies with thousands of employees who don't use any cloud services.

It can be done, and contrary to marketing, it's probably cheaper and more reliable.

What code is needed to make a decision to go with a smaller provider instead of AWS?

No, it's 100% appropriate. Anyone can have opinions on anything, but frankly, most of them have little relevance to reality. Their use of the word "expert" is supposed to mean the person has knowledge or expertise that renders their opinion on a subject substantially more valid and relevant than any regular person. That clearly is not the case here. If I wanted to know what a random person on the street thought about a subject, I could go ask one myself. The purpose of news organizations was supposed to be to better-inform people by getting opinions from actual relevant experts in a subject.

These people don't seem to have much ability to discuss relevant subjects like what the actual reliability of lower-tier hosting providers is, the value-add to business and iteration speed of having a variety of extra services (SQS, DynamoDB, VPC, RDS, managed K8s, etc) available, etc.

I think your third point is what I've had to attune to when criticizing cloud dependence. I think if your entire source of revenue is dependent on AWS then you should be prepared for 16+ hours of downtime per year. Individuals notice it more when something is down for hours but with good observability I am guessing the business notices it more when performance drags for the other 8742 hours of the year. Bursts of downtime per day can still be attributed to the device, wifi, ISP, or some other intermediary's DNS/BGP.

If your margins are so tight that 16 hours of downtime will bankrupt you then I think either: a) I have no idea how to run a business; or b) you have no idea how to run a business. I'm also biased because I love highly fault-tolerant, geo-redundant, durable systems much more than "good enough for this KPI".

> but with good observability I am guessing the business notices it more when performance drags for the other 8742 hours of the year

This is really good point that aligns with my experience. Today's event was LOUD and (compared to other incidents) long, but perhaps not really that long compared to the situation you describe that for most businesses is going to be more pernicious.

Business intelligence and analytics-type folks at $DAYJOB are _very_ watchful for the year-on-year deviations and even periods where the prediction lines didn't match up for even just a few hours.

I think all of that is mostly irrelevant. You don't need to pay a huge cost to avoid the small benefit, you don't need every service to be resilient to this, or any of that. You just need multiple different providers so that not everyone gets screwed at once.

Yes but we live in a highly anti-competitive monopolized world now. With more to come under the new admin.

It’s hard to think of anything less monopolized than cloud hosting. There are hundreds of providers.

For any business that matters, your choices are amazon, google, Microsoft, and that's about it.

I couldn't even name another provider except maybe Hetzner

The three you mentioned have over 60% market share which is why this article exists at all. Knowing what I know about cloud ifnra, anyone who is actually anyone is hosting on the big three. So it's not just a market share, it's market share + impact / importance.

You could also argue that YT is on GCP (to some level) and that would probably bump that number up much higher.

The vast majority of people hosting things on the internet are on these providers. But you get downvoted for pointing that out now.

Yeah right, and how many of them have any substantial customer base compared to AWS and Azure?

There’s two or three gartner approved ways of doing things for fortune 500 ctos, and f500 wannabes.

It’s not a monopoly but it’s close.

But that would require companies to actually spend time and money testing and working with either a cross-provider multi-master-type system (with all the associated consistency headaches) or regularly test a functioning disaster-recovery/fallback system.

The time spent on that (let alone cost, for companies with large amounts of data) far outweighs the cost when a single region has an issue of today's scope. And you said it yourself, it's a 'small benefit'. Small benefits sound like exactly the things not worth spending time or money on.

For as much as many companies have had issues today, the daily reality is that these same companies haven't been having issues all the rest of the time (or this wouldn't have felt so shocking) and are likely to be okay with an outage of this scope (plus, everyone's too busy making noise about the issues to be working normally).

There are multiple different providers, with nothing artificially limiting their use. Also idk what's so bad about Fortnite and Snapchat going down at once instead of it being staggered.

> - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words.

NO. From their own reports, clearly AWS is too centralized and dependent on a specific region (us-east-1) and a specific service (DynamoDB). This has been observed for well over 10 years. Why do they stay in this centralized architecture? Cloud services need much higher standards than the average corporation. Just look how they took down 2000+ services for many hours.

[1] https://health.aws.amazon.com/health/status

I don't deny that an incident of this scope should prompt a serious technical and process review (and as you describe it, it sounds like this is long overdue), however how often does this kind of thing not affect 2000+ services? Companies should be tracking the time they don't have issues as much as the time they do in order to actually understand if they'd be better off elsewhere.

And to be clear, I'm not at all arguing for the monopolisation of cloud providers, only stating that it's easy to point from far away and say 'This is bad' while simultaneously not doing anything to understand the cost and make that change that you say is important, because it's actually costly (in many dimensions) to do.

Even wearing my ex-AWS hat and understanding to some degree the internal complexity of these services, I too am boggled that foundational stuff is still out of Virginia and not a separately operated global region for the subset of control-plane dependencies that can’t be refactored into tolerating eventual consistency (such as parts of IAM).

We always used to talk a lot about minimising blast radius and there’s been enough time, and enough scale, to fix it.

Nevertheless the Guardian’s choice to label self-promoting policy wonks as “experts” is a cringe-inducing reminder that journalists don’t know anything about anything.

> Knows how much time national newspapers spend actually talking about the importance of multi-region/multi-cloud redundancy

For the record, multi-region redundancy is moot, and I can't stress it enough. It is not the first time that on the surface it looks like a single region but in fact services in multiple regions are affected.

And multi-cloud hot standby can be terribly expensive, unless your infra is very simple. And it's not easy to get it right either until you planned for it from day one.

Um.. you don't need to be an expert in security, comp.science or economics to know that putting all eggs in one basket may not be a great idea as introduces one giant systemic target. If anything, regular people here are uniquely qualified to say something along the lines of:

Oi, this is ridiculous. Maybe more things should be ran locally..

FWIW, it was instructive to me as to which companies were not able to function today.

> - Knows that genuinely 'critical' services (i.e. health) should be designed to account for this

yeah but aws advertises as "trust me bro I won't go down for 99.99999%"

I've seen a lot of gov proposals using aws to 'get away with downtime management'

These are Guardian 'experts' so can be safely ignored.

maybe your VC overlords need a reality check?

Because the experts have no say in policy. The only people who have a say are the people bribing (sorry I mean "lobbying") Congress. And even they have very little say because Congress is currently on a hot streak of doing absolutely nothing.