Hacker News Reader

Comment by katzgrau | Hacker News Reader

I was on a call with an investor and an employee mouthed to me, silently, "everything is down!"

Immediate hot flash.

After I got off the call (thank god he had an appointment to run to), I checked it out. Our internal dashboards were all green so we realized it was a DNS issue pretty quickly.

Since we couldn't get into Cloudflare we searched Twitter and realized it was their issue and I stopped worrying.

One of the benefits of CF and other major internet vendors is that when they're down, you can kind of shrug it off and tell your customers to wait a bit. Not so if you're using a smaller/unknown CDN company.

View on HN

As a famous haiku once said:

It's not DNS

There's no way it's DNS

It was DNS

Honestly I think this is an underappreciated aspect of the dominant role of Cloudflare. If we were using a smaller CDN then our users would blame us every time it went down. But because we're using Cloudflare our users will quickly notice that sites much bigger than us are down as well and just think "eh, the internet is being weird right now".

Twitter has become the go-to dashboard to discover if there is a broad based outage when our servers are inaccessible. In the last month, IBM data centers and now Cloudflare

Anytime Cloudflare is experiencing issues, I immediately stop what I’m doing and direct customers to the Cloudflare status page and take a nice break.

Why did you stop worrying?

You have a bigger name to blame. It is much easier to tell customer Amazon is down, or Cloudflare is down, rather than explaining something that customer dont want to hear.

And if everyone else on the AWS / CF is down as well, then it is no one's fault. We all keep calm and just wait it out.

Your customer doesn’t care, as they don’t have business with cloudflare. Your fault is in not having plan b.

They're right, though. Some companies are large enough to give you credibility cover during an outage.

It's the same as the old phrase about no one ever getting fired for buying IBM.

Yes and these sort of excuses applies to ALL industries. Any Sales people or Support Staff should already know this. So I am often surprised people are arguing about it.

At the end, your customer, assuming it is a SaaS, just need an excuse for "their" customers or the End User. And there is nothing better than a big household name's fault so no one get the blame.

So the whole thing is written off, everybody is in the clear, and everyone can get on with their other business. :)

Well many people have the ability to see through marketing BS, and these types of excuses are exactly that, marketing BS

I often feel sad for people that have not developed the critical thinking skills needed to look past marketing to where the realm of logic and reason resides

Patronising much?

The simple reality is that if your site is down because AWS is down then it means a lot of other sites are down too. Which means you don’t look anywhere near as bad as you would if you were the only site not working.

Logic and reason is all well and good but human perception is a very real thing that businesses need to keep in mind. It isn’t always entirely logical.

I would like to know of a time where all of AWS was down, every region, every data-center, everything down

At most one region was impacted, and it is easy to be multi-regional in AWS, in fact that is kinda of the point

Further this comment is in service of the moronic axiom of "No one ever got fired for buying <<insert large company>>" my response to that has always been and will always be "sure they have and they should"

It is simply not true that buying AWS, IBM, Cisco, or any other large vendor is complete insulator from all responsibility to maintain reliable systems nor should it be

Any administrator or developer that is making buying choices based on that is not an person I would like to ever work or do business with

“AWS is down” is phrasing shorthand, it doesn’t mean every single service they offer is nonfunctional. It means that for the vast majority of people it is not doing what it is supposed to do. Again, it’s not a technically correct statement but it is how humans talk.

To extrapolate out to the original point: if multiple top ten traffic web sites are having issues (and this has happened once or twice in the last few years due to AWS or other cloud issues) then your site being down is less notable in customers minds.

again that depends on who your "customers" are

The grandparent was talking about a SaaS service so then I would expect the customers to be technically minded people.

If you are selling to masses then sure, but if you are selling me a Line of Business SaaS service then no I do not care if facebook and reddit was down, that is not relevant to how the SaaS product should be running

I think that is a narrow definition of SaaS. But I understand your point.

For instance Basecamp could be down due to AWS. And Basecamp ( and in this case Hey as well ), but are SaaS. And their customer may not always be technically minded. Given the usage of these tools, while inside tech circle, are not only used by technical people.

And At the end of the day it is all about trade offs.

I also dont think anyone ever make a purchase decision purely on brand or not fired for X. For example, despite AMD offering lower price and offer more core and performance. Server Vendors hasn't all switched to AMD at once. In fact, Intel Server still has months of backlog order to fill. This isn't simply because Intel is better connected with Vendors, it is the fact most of those End user / customers are still demanding Intel CPU. Because it is well tested, with more specific libraries, tools, guarantees and support. Many of these factors cant be quantitatively measured, and therefore would only be judged when the final price difference are shown. In this case No one gets fire for using X is another phase for if it aren't broke, dont fix it.

I believe Xeon vs Epyc is more to do with the fact that migration from Xeon to Epyc is a disruptive process both at the hypervisor and guest level. Server hard ware changes today is

Install new host -> Migrate VM;s live to new host -> Shutdown old host

Zero Downtime

This is simply not possible with a Xeon to Epyc Migration which requires downtime, as well as testing of the guest to ensure nothing weird happens

it is a prime example of Vendor Lockin.

IF there was away to live migrate with zero downtime Epyc would own the datacenter today

Wish it were so simple, but alas, fair or not, this is a real and prevalent bias in the world.

I’ve had colo equipment that ran with 5-nine uptime for years eventually get unlucky and be down for an hour and it was “all my fault”. Switch the service to Amazon which achieves much worse uptime, but now it’s “well if Amazon is down, what can you do?”

Frankly when something seen as core internet infrastructure goes down, the measly SaaS companies pretty reasonably don’t take any blame.

To be clear, often these are services where there isn’t the engineering budget, nor honestly the need, for multi-cloud and geographically distributed multi-master services.

Agree, but it comes across as we don’t care. I worked for a major gaming network, and believe me, gamers don’t care why they can’t play. They want to play. Now. And I can’t imagine anyone on the team relaxing when something like that happened.

Yeah, luckily our customers are a little easier to deal with than that, despite the very real impacts to their businesses. You have to know/understand your customers in order to know when you can relax.

But the plan B in this case is switching nameservers, which we could certainly have done (and briefly considered), but it could be error prone and would take longer for those changes to propagate than it would for CF to fix the issue, most likely.

The best option was simply inaction, if that makes sense. There are times when not doing something is a better idea than doing something.

I was CTO at a SaaS company with a whole host of companies you've heard of relying on us as a core part of their business. They absolutely would and did accept "AWS was down, soz" as an excuse, where they were much spikier about fuckups where we were to blame.

Tangential: nobody got fired for buying IBM

I was in eng leadership at largest gaming network. Our customers (gamers) absolutely didn’t care about reasons they couldn’t play. It was always our fault as we were responsible for tech choices, not them. And I believe it is a way. If I’m selling you a service it is my responsibility to make sure it is reliable and not to blame it on something that is under my control (like cloud providers). Of course I can’t fix local internet going down, but I can make sure I’m not married to 1 vendor

I'd argue that you're probably holding yourself to a standard that is more/less unachievable in such an interdependent world. It's idealistic and idealism is a square peg in the funny shaped hole of reality.

Taking accountability and having backup plans are extremely important, but you simply can't remove every last shred of dependence. You eventually have to accept that there are things that are out of your control and may take you by surprise despite best efforts.

In web and online infrastructure pretty much nothing is out of your control except for two things: ISPs people use and domain name registrar you use for your domain name. And even domain name registrar centralization can be mitigated against by having multiple domains from multiple registrars and promoting different domains to different users and having backup communication channels to inform users about new domains in case something happens.

Other than that it's your choice whether to make your infrastructure dependent on a bunch of unreliable centralized SPOFs from big corporations or build highly available infrastructure relying on servers from many different providers running your own DNS servers with DNS routing, failover, etc. You will definitely beat Cloudflare's availability this way many times over.

And you will still be exposed to being blindsided by something out of your control. It's really only in your control of you can think of and plan for it ahead of time. And there will certainly be things that we don't consider. You can call that a failure but it happens all the time and it's reality.

What if a political event impacts you, for instance? A pandemic? A storm taking out a major data center? A weird Linux kernel edge case that only happens beyond a certain point in time? That only sounds ridiculous because it hasn't happened, but weird things like that happen all the time. There are so many unseen possibilities.

I understand that might sound unreasonable or facetious or like I'm expanding the scope.

The point is, the more confident that you've built something that has no SPOF the more exposed your are to the risk of it, because one probably does exist.

Honestly, you are not making any sense. This is not how engineering works. If you design for resilience, you get more resilience and you build confidence as you see the evidence how the system works in real world. Furthermore, with resilience you have to always cover all risks, it's just that you don't immediately reach fine granularity of decisions that don't trigger failover to servers in different countries, you improve granularity as you learn from actual operations and modify your designs accordingly.

I remember when I first deployed DNS routed system it was too reactive, constantly jumping between servers, monitoring was too sensitive, it didn't wait for servers to stabilize to return them into the mix and exponential backoff was taking servers out for far too long. But even given all that it was still able to avoid outages caused by data center failures and connectivity problems.

It does make sense, and it's paradoxical, I know.

> If you design for resilience, you get more resilience and you build confidence as you see the evidence how the system works in real world.

You simply can't foresee or eliminate all risk. This is referred to as "the turkey problem." It's not my idea, but one I certainly subscribe to.

https://www.convexresearch.com.br/en/insights/the-turkey-pro...

The whole idea behind resilience is to cover unforeseeable risks, the turkey problem just doesn't apply here. I would even say if the system doesn't solve the turkey problem it cannot be called resilient. And high availability without resilience is not practically possible.

> The whole idea behind resilience is to cover unforeseeable risks

Speaking of things that don't make sense... if it's unforeseeable, one will have a difficult time adequately preparing for it

It's not difficult, it's just different. It's the difference between predicting that a truck might crash into a data center and building concrete wall around it, and designing a system in a such way that users only ever resolve to servers that are currently available regardless of what happened to some of them in a data center that had a truck crashed into it.

... and after you've solved for the truck problem, you have a potentially infinite list of other things to plan for, some of which you will not foresee. And of course, there's probably an upper bound on the time you can spend preparing for such things.

Famous to the point of being a cliche, the titanic was thought to be unsinkable, and I would have a similarly hard time convincing the engineers behind the ship's design to believe otherwise.

The level of confidence you're displaying in predicting the unforeseeable is something you may want to take a deeper look at.

You are missing the point. Solving the truck problem is exactly what you shouldn't do, well, at least until your system is resilient. Because it could be something entirely different, it could be law enforcement raiding a data center and your wall around it won't protect it from them. So instead you approach the system in terms of what it has to rely on and all possible states of the thing it has to rely on. Which maps to a very small number of decisions. Like whether a server is available or not. If it's not available it really doesn't matter which of the infinite things that could happen to it or to a data center it is in actually did, you simply don't return it to users if it's not available and have enough independent servers to return to users in enough independent data centers to achieve specific availability. It's really not difficult.

I understand that most of those leetcode corporations don't care much about resilience, likely even incapable of producing highly reliable systems, and may give you a false impression that reliability is something of an unachievable fantasy. But it's not, it's something we have enough research done on and can do really well today if needed, we are not in titanic era anymore.

I have high confidence in these things (not in "predicting the unforeseeable"), because I've done them myself. My edge infrastructure had like half an hour of downtime total in many years, almost a decade already.

I am a customer of a SaaS product that use AWS, and I never accept "AWS was down" and should the vendor use that I start looking for new vendors

haha good luck with that, the vast vast majority of saas products are not multicloud, and nor should they be (cost / benefit tradeoff)

they should be multi-regional. If they are only in US-EAST and US-EAST is down well yes I blame them for that.

show me a time where all of AWS was down in every region all at the same time

The point is that the customer is likely using a dozen other sites that have also been hit by cloudflare outage, so they care less that your one specific site is knocked down. The customer immediately knows the blame is not on you when they notice how widespread the outage is.

Because they don’t care about their users. Last time dns quit on us (and literally half of the internet went down), we ended up developing a backup plan.

Not OP but likely because there was nothing they could do.

Reply on news.ycombinator.com