Why Twilio Wasn’t Affected by Today’s AWS Issues
twilio.com
twilio.com
nginx/0.9.2
You really want to make sure your shit works before you go boasting about how well it works. :)
EDIT: seems to be working now :P Interesting article once I got over the irony of it not working.
The Twilio Engineering blog is hosted off an external Wordpress site with a single IP that's forwarded from ngnix load balancer pool. Since the load balancers assume that the external service can fail, they won't tied resources blocking access to other parts of the site.
Hope you enjoy the post :)
-Evan Twilio.com
jdyer@aleph:~ [git:master] <ruby-1.9.2> » host api.twilio.com api.twilio.com is an alias for public-vip374d1ca4e.prod.twilio.com. public-vip374d1ca4e.prod.twilio.com is an alias for ec2-174-129-254-101.compute-1.amazonaws.com. ec2-174-129-254-101.compute-1.amazonaws.com has address 174.129.254.101 ----
jdyer@aleph:~ [git:master] <ruby-1.9.2> » host www.twilio.com www.twilio.com is an alias for public-vip29c4ab3d.prod.twilio.com. public-vip29c4ab3d.prod.twilio.com is an alias for ec2-174-129-253-75.compute-1.amazonaws.com. ec2-174-129-253-75.compute-1.amazonaws.com has address 174.129.253.75
http://online.wsj.com/article/SB1000142405274870457070457627...
-Evan Twilio.com
The new rent-a-bike scheme in London has POS terminals connected to the central system via bits of string and/or cellular modems. Every now and again these links fall over or the central system becomes unresponsive.
If you are attempting to get a bike (with an active card subscription) you drop your card into the terminal and it prints you a release code that lets you take a bike.
Unless the system is down... in which case it still reads your card, and then sits there and shows you a spinner for 5 minutes.
You can't walk away during this time, because if you do and the link comes back up it'll print a release code which anyone can use to take a £300+ bike on your account.
If you do stick around and try again? That'll be another 5 minutes which you could have spent walking to the next bike dispensary.
I think that timeouts are one of those things that you can only tune really well when you use the system in a live environment and see how well things work. In this case a higher transaction failure rate would be vastly better than a 5 minute time out - on other systems not so much.
[UPDATE] A central theme of the recent AWS issues has been the Amazon Elastic Block Storage (EBS) service. We use EBS at Twilio but only for non-critical and non-latency sensitive tasks. We've been a slow adopter of EBS for core parts of our persistence infrastructure because it doesn't satisfy the "unit-of-failure is a single host principle." If EBS were to experience a problem, all dependent service could also experience failures. Instead, we've focuses on utilizing the ephemeral disks present on each EC2 host for persistence. If an ephemeral disk fails, that failure is scoped to that host. We are planning a follow-on post describing how we doing RAID0 stripping across ephemeral disks to improve I/O performance.
What percentage of the various pools were affected by the outage?
I'm more curious about the hourly rate.
If you have a pool of 30 instances and only 3 are accessible, are you still being charged for all 30 plus the additional 27 you need to bring up?
Ditto for any web 2.0 we-are-a-fancy-shared-commenting-blog service, or anything that is fundamentally time based aggregation of information. Do database replication systems just not handle the concept of working with temporary gaps in the data?