I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs)
In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-)
The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues.
- Avleen Vig, Staff Operations Engineer, Etsy
Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals.
In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well.
Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.