Thanks!
How do your watchdog instances monitor other hosts? Do they track usage/load via snmpd, or something else?
Also, are they directly calling the EC2 API to launch or shut down hosts, or do you have an in-house deployment system?
How do your watchdog instances monitor other hosts? Do they track usage/load via snmpd, or something else?
Also, are they directly calling the EC2 API to launch or shut down hosts, or do you have an in-house deployment system?
If I was doing it over again, I'd just use Amazon's auto-scaling features for all of this. At the time we built this, EC2's auto-scaling didn't support some of the features we needed. Since then, they've made it a lot easier to do things like set up a repeating schedule for auto-scaling, rather than using metrics.
We only have one EC2 AMI that we use for all of our servers. That AMI is pretty basic; it only does enough to connect to our Puppet configuration management servers. Puppet then configures the boxes as web servers (or databases, or...) and adds them to the appropriate load balancer.
The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group.
How did you decide on the right number of instances? The article mentions 20%--is that based on latency, a cost-saving target, or something else?