Hacker News Reader

Comment by emrodre | Hacker News Reader

Their status page (https://health.aws.amazon.com/health/status) says the only disrupted service is DynamoDB, but it's impacting 37 other services. It is amazing to see how big a blast radius a single service can have.

View on HN

It's not surprising that it's impacting other services in the region because DynamoDB is one of those things that lots of other services build on top of. It is a little bit surprising that the blast radius seems to extend beyond us-east-1, mind.

In the coming hours/days we'll find out if AWS still have significant single points of failure in that region, or if _so many companies_ are just not bothering to build in redundancy to mitigate regional outages.

I'm looking forward to the RCA!

I'm real curious how much of AWS GovCloud has continued through this actually. But even if it's fine, from a strategic perspective how much damage did we just discover you could do with a targeted disruption at the right time?

The US federal government is in yet another shutdown right now, so how would some sub-agency even know if there were an unplanned outage, who would report it, and who would try to fix it?

Gov't IT is mostly contractors, and IME they do 99% of the work, so... That might be your answer.

AWS engineers are trained to use their internal services for each new system. They seem to like using DynamoDB. Dependencies like this should be made transparent.

Not sure why this is downvoted - this is absolutely correct.

A lot of AWS services under the hood depend on others, and especially us-east-1 is often used for things that require strong consistency like AWS console logins/etc (where you absolutely don't want a changed password or revoked session to remain valid in other regions because of eventual consistency).

Not "like using", they are mandated from the top to use DynamoDB for any storage. At my org in the retail page, you needed director approval if you wanted to use a relational DB for a production service.

> Dependencies like this should be made transparent

even internally, Amazon's dependency graph became visually+logically incomprehensible a long time ago

Ex employee here who built an aws service. Dynamo is basically mandated. You need like VP approval to use a relational database because of some scaling stuff they ran into historically. That sucks because we really needed a relational database and had to bend over backwards to use dynamo and all the nonsense associated with not having sql. It was super low traffic too

It's now listing 58 impacted services, so the blast radius is growing it seems

The same page now says 58 services - just 23 minutes after your post. Seems this is becoming a larger issue.

When I first visited the page it said like 23 services, now it says 65

74 now. This is an extreme way of finding out just how many AWS services there really are!

82 now.

Looks like AWS detonated six sticks of dynamite under a house of cards...

Up to 104 now, with 33 services reported as having issues that have been resolved.

i'm surprised / bothered that the history log shows the issues starting AM 10/20 -- when they seemed to have started around midnight 10/19

Reply on news.ycombinator.com