Hacker News Reader

Hacker News

Comment by jrs235 | Hacker News Reader

Parent Full thread

jrs235

I checked their health status page. All is good. /s

https://downdetector.com/status/aws-amazon-web-services/

View on HN

They did add an update, faster than last time:

"7:42 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region."

https://status.aws.amazon.com/

Edit: They added US-WEST-1:

"7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region."

Edit: Found root case, maybe?

"8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery."

"8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-2 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery."

Too bad I am unable to load the status page due to connection timeouts, so I can't see the updates.

someone tripped over the fiber run i bet. Or, a cleaning person unplugged a router to plugin a vacuum (that actually happened but to a minicomputer iirc)

Unfortunately the vacuum, a shiny IoT connected appliance, didn't work because AWS was down

Usually the problem is "an idiot with a digger".

nah man, it's never the digger that's the idiot. it's always the project manager that told the digger where to dig. just like it's never the dev's fault as the PM made them do it. /s

No way a cleaning person can do that in a datacenter.

I hope that their infra is not that unstable

It's interesting that west-2 was quicker to create the incident (despite the issue starting a bit later there, at least by our experience), and while they both "identified" at the same time, west-2 also waited longer to call it resolved.

I assume there are different teams responsible for each, is the west-2 team just more on top of things?

West-2 also launched many years after us-east-1, so less legacy to deal with.

1.US-East-1 wasn't involved today.

2. They don't really have much "legacy" stuff to deal with since they likely turn over racks quickly across their whole fleet and software deployments should be standardized, so any US-east-1 flakiness has to do with the fact that its where amazon houses their control planes often.

There's at least one AZ in East-1 that doesn't support nitro, and that's been around for 4ish years now...

I agree in principle, but clearly something is hobbling them because of (probably) legacy stuff

The issue is not specific to the US, same issues in Europe. Also, it seems not only AWS experiencing issues. Unless Google is hosted on AWS haha...

Yes, it could be network peering related. But there's definitely a lot of us-west-1 and us-west-2 users complaining and people saying that us-east-1 seems fine.

Seems to be resolved now. And seems they hid / took away any mentioning of possible issues. Sigh.

It's still there now, on the top of the page, just marked resolved:

us-west-1:

7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region.

8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.

8:10 AM PST We have resolved the issue affecting Internet connectivity to the US-WEST-1 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.

us-west-2:

7:43 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region.

8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-2 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.

8:14 AM PST We have resolved the issue affecting Internet connectivity to the US-WEST-2 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.

That is a shame. Anyone coming in after the fact to investigate an outage or glitch with their systems will need to look harder to find a known AWS outage. We can’t assume everyone looks at HN.

Practice makes perfect

So it is down again.

Ok, so it can't be down then. This is proof!

Yep, when it loads, it's all green. "nine nines!!!"

I thought that sounded ridiculous so I did the math and 99.9999999 uptime allows for 1.314 _seconds_ of downtime every 1000 years. It would take approximately 2.7 million years to acquire just an hours worth of allowable downtime, that's how long it takes light from the second nearest spiral galaxy and farthest visible object to the eye in perfect conditions [1]. Within a single quarter of a year, that's 328.5 μs (microseconds) or about 1200 blinks of an eye [2] or about 3 times faster than a typical electric capacitor camera flash [3], also approximately, and interestingly enough, less than 1% of my current ping to my ISP let alone Amazon's servers.

So yeah, having done that I now understand that it was probably a joke but it really puts into perspective just how ridiculous things can get with a few 9's.

[1] https://earthsky.org/clusters-nebulae-galaxies/triangulum-ga...

[2] https://www.verywellhealth.com/why-do-we-blink-our-eyes-3879...

[3] https://en.m.wikipedia.org/wiki/Flash_(photography) (wikipedia's won't let me deep link on my phone, it's in electronic flash section under types)

emphasis on when

60% of the time, it's all-green 100% of the time

Down detector is just a statistical page, it does not actually detect downtime, and is in no way aws's status page.

What does downdetector run on?

User reports — i.e. the number of people who google “is X down” and then click a Down Detector link.

It’s a clever way of getting reasonably accurate data very quickly and easily, though it does have it’s flaws — the data is pretty noisy and users often attribute outages to the wrong service (e.g. blaming their ISP or Microsoft or something when YouTube is down, or vice versa).

I would guess the user is asking what are down detector's dependencies... E.g. can their website function I'd us-east-2 goes down? Or a GCP equivalent? Or are they on a self-hosted server ? What would cause the metrics to be "off"

Reply on news.ycombinator.com