Hacker News Reader

Comment by flareback | Hacker News Reader

fta: "The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet."

I read that phrase everytime something like this happens and yet we all still rely on the same handful of companies.

View on HN

The answer is that before the existence of those companies to rely upon, people didn't rely upon them, they just accepted the lag, or they hand hacked the same globally distributed approach on their own, and it sucked for them, and it wasn't too great for users either. CDNs are big because that's what their function is, to reach the world and absorb traffic spikes, and take the complicated business of distribution of edge servers out of the hands of the people who just want to run a website.

The trade off here is intrinsic and accepting the risks of big CDNs is the right answer.

Exactly. The question isn’t “a few CDNs” vs “Many CDNs”. There are too many economies of scale. It’s really CDN vs Not? (Roll your own just isn’t feasible except for a half dozen of the largest tech firms)

Well, you could use two CDNs instead of just one. But that costs money.

I think it will continue to happen. People and orgs, when picking a service, don't have an incentive or a way to ask around to see what percentage of similar-service users are using that particular service. People and orgs will always flow towards cheap/popular/well-known services.

On the other hand, those handful of companies could be asked to structure their services so that an outage only affects a portion of customers and not all their customers. However! That would be more inefficient for them, and more expensive, and that cost would cause the people and orgs mentioned earlier to just flow towards the company that took those shortcuts.

There is an incentive to see what everyone else is using, so that you can use it too. Choose boring technologies applies to infra as well. Go with a big stable well known can or cloud provider instead of trusting your service to some fly by night startup etc.

That's because many such business have grown so large because they benefit from scale.

Some examples:

Social network: you only engage on one if your friends/family/coworkers are on the same network.

Search engine: needs to index "the whole Internet", which is less expensive per user if you have more users

CDN: works best if you have edge nodes everywhere, which is quite capital intensive, which is why you need many customers to distribute it over.

... and so on. We might not like it, but many of these quasi monopolies are based on fundamental economics, not (just) on the greed of the companies.

> Social network: you only engage on one if your friends/family/coworkers are on the same network.

This one isn't exactly based on fundamental economics given that federated social networks exist. Email has similar network effects and is not centralized.

I think this argument could be made for many of these.

None of the federated social networks seems to have reached the scale of the biggest centralized social networks.

Which leads me to believe that economics and incentives favor big, centralized social networks.

If you're making a purchasing recommendation for your company, do you want to tell your boss that you're recommending not going with your best option, or even 2nd, but that your company should use the 4th or 5th best CDN as a way to diversify the Internet? Seems pretty altruistic, but not a great way to keep your job.

It's the tech equivalent of "thoughts and prayers" isn't it?

There are at least 5 major CDNs out there. CloudFront, Fastly,Cloudflare, Akamai, Google CDN. You can use more than just 1. Shopify uses two. Akamai and Fastly.

If you use them as pure file-serving CDN, then sure. But once you start adding extra logic, headers, routing, etc. the features don't fully align. Or you need to keep to the minimum common featureset.

Does anyone know if Azure/Google/Amazon can provide some 'multi-cdn' setup out of the box? The way to change these points of failure is for the big boys to change their defaults.

Do it at the DNS layer. Route53 has failover support out of the box that should work for it. You can setup a monitor and it will switch dns entries on a failure.

It is getting increasing tricky to have enough redundancy at a basic level to avoid a major player outage from affecting you. For example, you would probably want at least one authoritative DNS server that isn't either of your CDN providers. And knowing some details about how these players sometimes use each other, like that Google's Firebase uses Fastly.

Are you bigger than a major player is the question I'd be asking. Maybe risking it is fine.

I don't know that size of your operation is the right metric to gauge whether to bother with this. If 100% of your revenue, for example, is from online sales, it might be worth it even if you're small. But yes, it's often not worth it.

I agree, it's just for some DR scenarios there's only so much you can do. And 'the internet is down' is hard to plan for. If CNN is offline due to some outage and you're a smaller enterprise then are people really still doing online eCommerce stuff, or are they waiting for their favorite sites to come back up as a signal that things are back to normal.

But we're talking about what was a 1 hour outage. Does it make sense to spend more than 1/8000 of your revenue to avoid an hour per year esp when you will never lose an hour of revenue from being down for one hour because many customers come back to buy the thing they were going to buy anyway.

So, after multi-cloud now we need to go multi-CDN? Half joking here, it's actually a good idea although probably it's not worth the cost. I think GitHub (at least from my casual looking at that behavior during the outage) nailed it because they must have some kind of active/passive CDN config. They were affected by the outage but after a few minutes (less than whole Fastly outage duration) they were serving assets again.

Would a smaller CDN provider have no outages at all?

No, but it would take down less sites if it does have one.

Is that a feature?

If we have 12 small cdns have 12 outages in a year (combined, 1/year each), each time bringing down 10,000 websites, is that better than 1 large cdn having one outage during the year bringing down 120,000 websites?

If I'm the website owner I think I prefer the latter, my customers blame the cdn instead of me. If I'm the cdn owner I definitely prefer the latter, more customers to amortize my costs over.

From a customer point of view diversity is far better it's great - if Sainsburys is closed for some reason I'll go to Tesco.

Certainly don't want a situation where all the shops are closed.

It may be more acceptable to have more frequent outages if the impact radius (number of websites or services impacted) is smaller.

Obviously that isn't an honest comparison. If you're asking whether or not 10 small CDN providers could provide a more robust, higher quality service with more uptime than one large CDN provider, then I think the answer is probably "yes."

There's kind of a race-to-the-bottom[0] wrt. decentralisation.

It'd be better for the internet as a whole if we don't always pick the most popular (so when your email's CDN goes down you can still communicate on chat, when CNN goes down you can still read BBC). But as an individual I have strong incentives to pick the one everyone else picks, because that's presumably the most stable/documented/lowest cost due to volume.

[0] https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

One of the side-effects of the efficiency obsession that capitalism generates, is the constant strive for centralization.

We took a technology stack designed to survive nuclear attacks, and turned it into something where a single bug can take down half the services on it. Why? Because on the flip side, a single improvement in the centralized service can automatically cascade to all the businesses using it.

Efficiency is a double-edged sword.

Reply on news.ycombinator.com