Ask HN: AWS SQS down in us-east-2?

7 points·jpetrucc··5 comments

Seeing 503's in both the AWS console, and in our logs.

[edit]: It also appears now that the following services are all experiencing issues in us-east-2:

- SQS

- Lambda

- CloudWatch

- IoT

[edit]: As of 11:18 AM PDT, it appears that SQS and CloudWatch are back up.

View on HN

Yes, it is throwing 503s and we're also getting general network failures in other calls to the SQS APIs. As is typical, not a peep from AWS or any indication of what is going on.

[Edit 1] And we're seeing the issue in us-east-1 too FYI

[Edit 2] They are now reporting the issue in us-east-2 with SQS and Lambda

[Edit 3] After looking harder it appears I was wrong about us-east-1. I thought we were seeing failures there but it does seem to be confined to us-east-2 only.

[Edit 4] AWS is reporting it is back up and it appears the errors stopped for us at around 1:10pm Central

A separate questions. Are their any common practices for recovering the system from message bus failures (such as SQS in your case).

Seeing your post made me realize if SQS goes down, a lot of random things will be happening in my application.

I'm also curious about this - I've not put much in place on AWS for SNS or SQS failover. I found the following Stack Overflow post [0] that has given me some ideas about how to make our services do a pseudo-failover for SQS

[0]: https://stackoverflow.com/a/39395654/6650640

https://status.aws.amazon.com/

Unfortunately it normally takes them >30 minutes to start reporting issues on their status page, so it's hard to know if it's just us or if it's widespread.

Reply on news.ycombinator.com