AWS: Network Connectivity issues affecting EC2 in US-EAST-1 | Hacker News Reader

AWS: Network Connectivity issues affecting EC2 in US-EAST-1

status.aws.amazon.com

55 pointsalanbyrne31 comments

Again with the green tick with an 'i' icon as a status, rather than a yellow/red icon, jeeze.

Had a bunch of timeout alerts. Some machines of an application server array have the packet drop issue. ELB says that everything is peachy. Folks, we're experiencing yet another "EC2 flavored SNAFU™".

Initial though was: f#*k, I ran out of network I/O, since EC2 simply states "low, mid, high" as performance specs, therefore some proper planning is out. Turned out that with load avg under 0.02 on all machines, the I/O wait wasn't to blame. Average response time, per Pingdom, went up from 170ms to 450ms. New Relic isn't happy either. I guess we should all thank Amazon. Again.

I see the same thing. Pingdom reports higher response-time, but no downtime (meaning no alert). Also no alerts from AWS Cloudwatch. I first became aware of the issue when internal api-tests started failing at 9:56am CET. I see users accessing the site, but I don't know how many it's failing for.

No issues in the internal EC2 network. At least, none that I could find. I guess that's the reason why ELB doesn't shift any traffic. The whole issue seems to be on the Internet facing network. Failing routers, maybe.

Pingdom still claims 100% uptime, but New Relic (which includes an equivalent pinging service) reports downtime from time to time. Around 25 timeout alerts into the last couple of hours.

Down vote? Really? I guess I should be thankful that I didn't have to explain this to a client: http://i.imgur.com/URf0H.png

Health Dashboard updated the status to "Resolved":

5:17 AM PDT Between 12:51 AM and 4:52 AM PDT we experienced elevated packet loss affecting instances in the US-EAST-1 region. Some of our APIs also experienced increased error rates and latencies. The issue has been resolved and the service is currently operating normally.

Bullshit, I am still down.

This started a little before 01:00 PDT (08:00 UTC), so we're approaching the 3 hour mark now. FWIW, that's about 1h15m before there was any sort of indication of a problem on status.aws.amazon.com

This got me wondering for quite a long time: I'm pretty sure almost all EC2 issues happen on the US east region -- why is that? Is it because it's the most used region?

Think so... it's the cheapest, and is the default choice. Sounds like it would have an order of magnitude more usage (and therefore problems) than the other regions.

The US-East-1 is also spread across more than 10 datacenters, which doesn't make things easy. Might not be true for other, more expensive, regions.

My site is affected. I don't know how to handle this situation because the issues affect whole region.

You should have duplicate infrastructure in another region which you fail over to automatically or manually.

We are going to set this up for our app, however there's a limit to how useful it is when you are reliant on services such as MongoHQ/PubNub (in our case) that are also affected!

Well, it's not going to be cool or best practice to use cloud services for your startup for the next week or two. Fortunately people on HN will then forget until the next AWS outage and your setup will be fine again.

Duplicating across regions is not well supported by AWS unfortunately. Any idea doing that without completely rebuild the infrastructure in an other region?

I assume you are using the AWS dashboard to manually deploy instances and setting them up by hand? First step is to get your infrastucture to the point where it deploys and scales up and down by itself. Once you manage that, moving to or keeping hot spares in another region is pretty easy.

Look into automated deployment tools like Foreman and configuration management tools like Puppet.

You assumed right.

We are learning the hard way that the EC2 Availability Zone separation is not enough, and EC2 is lacking some key features offering multi region tools.

Thanks for the hints, I'm going to check them out.

I echo this... although I don't personally use Foreman. You can do scaling type stuff without the tool and I just use POP (plain old puppet). For me personally it took longer to get on the config-management bandwagon but once I did I have never looked back.

Learning to use Puppet, Chef, Salt or similar will only bring benefits!

How would you recommend performing that failover, when Elastic Load Balancers only cover a single region? If you host your own load balancer it'll have to be hosted somewhere, and DNS failover seems unpopular [1].

[1] http://serverfault.com/questions/60553/why-is-dns-failover-n...

DNS failover is fine, even if its manually triggered. We are almost two and a half hours into this network event, so even if you set a 5 minute TTL on your records and it takes you 15 minutes to respond, you are still way ahead of the game.

The idea that some DNS resolver somewhere caches records longer than TTL is mostly a myth. This is the case rarely on some mobile networks and networks behind satellite uplinks, but they generally have larger network issues anyway.

Some of my us-east-1 servers are affected, others are happy so far, so it's not the entire datacenter. However, connectivity on affected servers has gone from just flaky to completely gone.

Odd... my site is on Heroku and there seems to be no trouble.

git push seems usable. :(

i can't git push to heroku...

Our app is also on Heroku and it did experience problems.

US-EAST-1 seems to have more issues than their other data centers... anyone know if this is really true?

It does, but not for nefarious reasons -- it (until the more recent history) the cheapest region in the world for AWS -- it was only right before Oregon rolled out that Ireland, US-EAST and US-WEST-2 all became the same price point, but for the 4 years prior to that, it was always the cheapest so that is where most customers rolled out most of their infra.

Now that the prices has normalized I think the load is distributing more evenly, but for historical reasons I think that region sees a lot more churn (starting/stopping/deploying/etc.) -- just more grinding on the hardware at that region that others.

rkalla's answer is good, but there's also the issue of the US east coast being a good place to deploy applications that need decent performance to the majority of the english-speaking world. Dropping your servers there puts them within reach of Europe without as much RTT as the US west coast.

Obviously the right answer is to deploy applications into multiple AWS regions, but that's not appropriate for every service's architecture unless built that way from the start or modified specifically to do so.

In addition to the other two excellent answers, there's another reason for the apparently higher issue count:

US-EAST-1 is enormous. It's now made up of 10 data centers with tens of thousands of machines (and associated infrastructure). There are thus more moving parts, and more points of failure. While Amazon builds a bunch of redundancy into its systems, "smaller" issues will tend to impact a larger number of users in US-EAST-1.

Looks like this is the reason I can't push to heroku either.

Reply on news.ycombinator.com