Parts of AWS are down | Hacker News Reader

Parts of AWS are down

33 pointsBob31237119 comments

RDS & S3 is down for me right now in US-East region and apparently some parts of Frankfurt are also down.

There own favicon is timing out: https://awsmedia.s3.amazonaws.com/favicon.ico

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

Where you see this is DDoS?

I'm not OP, but I heard that unofficially from somebody who works at aws.

I just got off a chat with an AWS agent:

"We are investigating reports of occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name."

From the phonecall with our AWS TAM and support reps

We manually updated our `/etc/hosts` file with the static ip for our S3 bucket(s) and that fixed the issue temporarily.

Glad to hear the issue is on Amazon's side. Our team panicked and tried to find every way to resolve this thinking it was possibly a misconfiguration at our end. I hope it's resolved soon. We're receiving complains from many users who are unable to upload their work due to this issue, I will let them know Amazon is on it.

Same here. At the time the status check on AWS was not reporting any issues and there was nothing written about it online. As a result I assumed it was an issue with my account.

I only confirmed it was on their end when I noticed the favicon on the AWS console wasn't loading.

Same here, I was running around like a headless chicken and wondering what I was doing wrong when elasticbeanstalk just wouldn't deploy anything. The fact that we doubt ourselves rather than thinking aws is down is a good damn indicator how reliable they are, luckily it happened during daytime (EST anyways)

The issue started happening much sooner, around morning time in EST, but it got worse as the day progressed. It looks like the Amazon engineers woke up and realized something was wrong and they updated their status page 5-6 hours late. But I agree, AWS is super reliable, it's unfortunate that this problem occurred and I hope it's resolved soon.

From https://status.aws.amazon.com/

Intermittent DNS Resolution Errors

We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.

Finally acknowledging the issue.

Seeing major issues here as well. All my modules were not working for the majority of the day and all images were not displaying. This is no doubt a Ddoss attack, I just don't see any other reason that the entire east coast AWS DNS servers would just stop responding all of a sudden. If the issue is not fixed by tomorrow our IT guy is gonna have a damn field day implementing all the temporary mitigation's.

Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.

https://status.aws.amazon.com/

Why does the status page say resolved when S3 is still down for parts of US?

status check: Tue Oct 22 22:50:22 UTC 2019

DNS resolution to AWS S3 and RDS is still buggy on other providers (DigitalOcean, etc as well)

https://status.digitalocean.com/incidents/1z3kmlvz69v6

https://developercommunity.visualstudio.com/content/problem/...

Having DNS issues with s3-us-west-2 as well.

Having issues here in US East as well.

Seeing the same thing

Reply on news.ycombinator.com