The OP says they can spin up a new bare metal machine in 10 minutes. It's likely the performance hit from running in a virtualized environment isn't worth shaving time off the 10 minute build.
I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals.
In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well.
Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.
For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs)
In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-)
The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues.
- Avleen Vig, Staff Operations Engineer, Etsy