Data Mining 3.4 billion Web pages for $100 of EC2
blog.luckyoyster.com
blog.luckyoyster.com
I'm sure you have all heard of the Chaos Monkey that Netflix runs? Well, I didn't actually even need to code a chaos monkey... All you have to do is run everything on spot requests. Eventually you will lose servers at unpredictable times because someone outbids you[1].
The typical spot request pricing is at least 3/10th of the price of an on demand. For instance a cp1.medium (4 cores 4GB RAM) costs $0.044 per Hour for a spot request. Compare that to on demand and it is $0.186 per Hour.
I bid $1 per hour for my spot requests across two zones in the same region. I group my servers and use ELB (Elastic Load Balancers) to route requests...
Typically, a spot request might last for about a week before it gets killed because the capacity isn't there. That's then when instances in my other zone takes 100% of the load temporarily. At this point, since I've lost an entire zones worth of servers, I have my auto scaling group fire up on demand instances until I can get some more spot requests fulfilled. Creating a setup like this took about a week, but the savings are enormous.
----
How is the data stored on this setup?
RDS(MySQL) handles all data that can be stored in a database.
Ephemeral Storage is used to store things that don't need to be persistant (ie transactional logs).
Sessions are managed through Redis.. If the redis servers die, then session handling is handled via mysql temporarily. (It's a lot slower but the mysql server is RDS so it's always running)
Elastic Block Storage volumes are automatically mounted to a single instance which is then set up as a NFS server->client in order to allow other servers to read from a particular mount point (ie.. A user uploads an image, and it's stored on the NFS mount. A different server reads the file and starts generating different dimensions, uploads all of the files to Amazon S3, and then deletes the original file on the NFS device).
The worst part about losing servers is when the memcached server dies, because I could lose weeks worth of cache storage. When this happens, I have to boot up several micro instances that take my "cache warming" list and basically start repopulating memcached again.
The entire system is designed to be redundant... I can kill every server and then run the initialization script to start up the entire stack. (It's basically lots of little cloud-init scripts[2])
I haven't used spot instance before, I am curious how you handle the termination gracefully? i.e. When the spot instance get terminated during the middle of transactions, e.g. uploading large file, writing to DB etc.
Have you found typical spot instance pricing to be significantly cheaper than reserved instances?
How do spot servers work (or better, how are they shutdown)? Can you subscribe to a notification prior to shutdown or it's like someone ran a shutdown -h on it?
Do you have redis saving to EBS then?
Regarding uploads of images, because users uploads are saved directly to the "uploads" EBS volume which is shared using NFS, even if the instance is in the middle of an upload to S3, I will know that it failed because I could do a simple query to check my database to see when a job was started, and when it was finished. Whenever my files are "syncronised" to S3, I then set the file in the database to "s3Synced=true". (The reason why I don't upload directly to S3 using the new signature/CORS feature is that I have to generated 6 different dimensions of each uploaded image and then also watermark)
If the server running the single EBS dies (this is a single point of failure for me, but could easily be avoided if I wanted to run something like gluster or even two EBS volumes on different servers (which I don't )), then file uploads are suspended temporarily. It's a bit pointless accepting uploads because although I could store them on each server that received the request, the batch processing of resizing them and watermarkering is performed by gearman workers which run on a small cluster of a handful of micro servers.
If a server is in the middle of a transaction and dies, well then you're kinda screwed. I'd suggest that if you know that certain traffic must not drop for any reason, such as payment gateways etc, then you should route traffic through a reverse proxy to an on demand/reserved instance. You can do this in nginx effortlessly.
I'll write a blog post about the entire stack next month if people are genuinely interested. Jeff Barr just dropped me an email actually (Hi Jeff, if you are reading this comment again, saying he would be interested in a full write-up).
Do you have data on how frequently your spot instances terminate gracefully (with time to finish their requests) versus how often they zap out?
Paying list price for any load that isn't mission-critical and needed immediately is insane.
In short, it's not a perfect supply-and-demand market, but it's interesting to see the details they found.
http://aws.typepad.com/aws/2011/08/run-amazon-elastic-mapred...