Hacker News Reader

Comment by mystcb | Hacker News Reader

Not sure it helps, but got this update from someone inside AWS a few moments ago.

"We have identified the root cause of the issues in the US-EAST-1 Region, which is a network issue with some network devices in that Region which is affecting multiple services, including the console but also services like S3. We are actively working towards recovery."

View on HN

That's a copy-paste, we got the same thing from our AWS contact. It's just enough info to confirm there's an issue, but not enough to give any indication on the scope or timeline to resolution.

Internally the rumor is that our CICD pipelines failed to stop bad commits to certain AWS services. This isn’t due to tests but due to actual pipelines infra failing.

We’ve been told to disable all pipelines even if we have time blockers or manual approval steps or failing tests

They finally updated the status page: https://status.aws.amazon.com/

Ahh good spot, it does seem that the AWS person I am speaking too has a few more bits other than what is shown on the page, they just messaged me the same message there, but added:

"All teams are engaged and continuing to work towards mitigation. We have confirmed the issue is due to multiple impaired network devices in the US-EAST-1 Region."

Doesn't sound like they are having a good day there!

I love how they are sharing this stuff out to some clients, but its technically under NDA.

Yeah, we got updates via NDA too lol. Such a joke that a status page update is considered privileged lol.

Reply on news.ycombinator.com