Hacker News Reader

Comment by mabbo | Hacker News Reader

Are the actual services down, or is it just the console and/or login page?

For example, the sign-up page appears to be working: https://portal.aws.amazon.com/billing/signup#/start

Are websites that run on AWS us-east up? Are the AWS CLIs working?

Anecdotally, we're seeing a small number of 500s from S3 and SQS, but mostly our service (which is at nontrivial scale, but mostly just uses EC2, S3, DynamoDB, and some basic network facilities including load balancers) seems fine, knock on wood. Either the problem is primarily in more complex services, or it is specific to certain AZs or shards or something.

Definitely not just the console. We had hundreds of thousands of websocket connections to us-east-1 drop at 15:40, and new websocket connections to that region are still failing. (Luckily not a huge impact on our service cause we run in 6 other regions, but still).

Side question: How happy are you with API Gateway's WebSocket service?

No idea, we don't use it. These were websocket connections to processes on ec2, via NLB and cloudfront. Not sure exactly what part of that chain was broken yet.

This whole time I've been seeing intermittent timeouts when checking a UDP service via NLB; I've been wondering if it's general networking trouble or something specifically with the NLB. EC2 hosts are all fine, as far as I can tell.

We're seeing issues with EventBridge, other folks are having trouble reaching S3.

Looks like actual services.

My website that runs on US-East-1 is up.

However, my Alexa (Echo) won't control my thermostat right now.

And my Ring app won't bring up my cameras.

Those services are run on AWS.

Now I'm imagining someone dying because they couldn't turn their heating on because AWS. The 21st Century is fucked up.

One of my sites went offline an hour ago because the web server stopped responding. I can't SSH into it or get any type of response. The database server in the same region and zone is continuing to run fine though.

Interesting, is the site on a particular type of EC2 instance, e.g. bare metal? I see c4.xlarge is doing fine in us-east-1.

It's just a t3a.nano instance since it's a project under development. However, I have a high number of t3a.nano instances in the same region operating as expected. This particular server has been running for years, so although it could be a coincidence it just went offline within minutes of the outage starting, it seems unlikely. Hopefully no hardware failures or corruption, and it'll just need a reboot once I can get access to AWS again.

I can't access anything related to Cloudfront, either through the CLI or console :

  $ aws cloudfront list-distributions

  An error occurred (HttpTimeoutException) when calling the ListDistributions operation: Could not resolve DNS within remaining TTL of 4999 ms

However I can still access the distribution fine

I see started EC2 instances are doing fine. However, starting offline instances cannot be done through AWS SDK due to the HTTP 500 error, even for Ec2 service. The CLI should be getting the HTTP 500 error too since likely the same API as the SDK.

My ECS, EC2, Lambda, load balancer, and other services on us-east-1 still function. But these outages can sometimes propagate over time rather than instantly.

I cannot access the admin console.

I'm getting blank pages from Amazon.com itself.

We've had reports of some intermittent 500 errors from cloudfront, apart from that our sites are up.

I can tell you that some processes are not running, possibly due to SQS or SWF problems. Previous outages of this scale were caused by Kinesis outages. Can't connect via aws login at the cli either since we use SSO and that seems to be down.

I wasn't able to load my amazon.com wishlist, nor the shopping page through the app. Not an aws service specifically, but an amazon service that I couldn't use.

EventBridge, CloudWatch. I've just started getting session errors with the console, too.

Using cli to describe instances isn't working. Instances themselves seem fine so far.

Interactive Video Service (IVS) is down too

Reply on news.ycombinator.com