What powers Etsy
codeascraft.etsy.com
codeascraft.etsy.com
I wonder in this case why, if any specific reason Etsy is not utilizing virtualization of one kind or another? Basically they're building dumb-boxes that seem a much better fit to have a good number of slightly beefier VM systems and just rebuild bad instances from images (this would be especially good since they're already utilizing Chef).
I know that as an e-commerce firm this has helped us substantially to move toward this kind of a structure.
http://37signals.com/svn/posts/1819-basecamp-now-with-more-v...
I think there are certainly specific web and db loads that aren't going to work as well under VM scenarios, but on the whole our experience has been positive. There will, I imagine, always be a slight 'slowdown' due to VM but with the gigantic push toward 'the cloud' right now, it seems to me one of the last things on the minds of most (and obviously Xen is highly used in that field, which is also what we use).
The bigger question I think is if you are using a highly CPU bound task which VM isn't always very good at. According to Etsy they are very CPU bound, so it makes some amount of sense.
Now, if you have 10 legacy apps on old P4 xeons and want to lump them all onto a sandy bridge server and save a bunch of power and rackspace, then separate VMs on one machine makes sense.
Basically, a cluster of 30 servers, all running exactly one VM doing the same thing is on-its-face silly. Why even have the VM then?
The Cloud(tm) excels at addressing some of these. You don't have to pay /as/ highly for staff to manage you machines and network (the provider does much of that for you). You can run smaller, cheaper instances much closer to their limits.
The downside is that you pay a premium to the provider (even if you are your own provider). Additionally you lose a great deal of opportunity cost. If you use the "excess" capacity on the weekend for hadoop jobs, and need to stop them because you have a large burst of traffic, you've hurt yourself. Your non-production hadoop jobs can also have unexpected and unintended impact on your production web servers, causing your users pain.
Given these things, if you focus is on keeping the best experience for users then you should split things apart.
The actual gain you get from cramming as much as you can onto one piece of hardware is much, much lower than you might expect. It ends up being easier just to get more hardware and dedicate that hardware for specific tasks.
- Avleen Vig, Staff Operations Engineer, Etsy
VMs shine when you're stuck with proprietary software that can't all be ported to run on the same kernel, then what you're paying for is a compatibility shim that's cheaper than another box. I don't see much sense for software you wrote in-house.
I think most shops with a high hardware need would probably have a mix -- things that scale well to the hardware (eg: database servers) might be set up as clusters of physical machines.
Less (performance) critical stuff like mail servers, bug tracking and time sheet appliances, backup domain controllers etc -- might be run as vms.
I agree that if you find yourself running a single vm on a physical box, you should probably reconsider how you are doing things...
Then again, all things considered -- if your (presumably) easy-to-mange vm based setup gives you enough performance -- why not keep it.
If nothing else as you upgrade hardware, if your workload doesn't increase exponentially, you will be able to consolidate on fewer servers. Consider a database service that in 2003 was set up master-slave on two physical pizza boxes. That could probably be run from two vms (on physically different boxes, for redundancy) -- with capacity to spare on either box today.
I think the performance hit for many people is overstated. Unless you're at the long-tail of CPU bound instances I imagine it won't become very apparent. That said, if I'm missing something/incorrect then someone please correct me!
Increasing densities through virtualization allows you to perform more discrete functions, but obviously reduces the aggregate capacity to do so. Etsy specifically has a goal of minimizing instance types, but virtualization would enable them go in a direction counter to their goals.
In my experience, the biggest weakness of virtualization is subpar I/O, and I would imagine that this would be an issue for Etsy as well.
Virtualization has a very real cost. If the performance hit (CPU & I/O) is (say) 20%, that's 20% of your capacity you pay to keep operational but don't get to use. The benefits don't outweigh that cost for every situation.
For example, let's look at some common web stack infrastructure: * Web servers * Database servers * Monitoring servers * Asynchronous task processors * Scheduled task servers (eg, cron jobs)
In all of these cases, we want the service on that hardware to be able to use 100% CPU. For example, it doesn't make sense to get N web servers, and then run M virtual instances with apache on them, when each instance will max out its available resources. Just run N dedicated web servers :-)
The same is true for pretty much everything else too, sometimes for different reasons. Eg, you don't want your database servers or monitoring system to be impacted by multi-tenancy issues.
- Avleen Vig, Staff Operations Engineer, Etsy
A bit about the dev infrastructure here: http://codeascraft.etsy.com/2012/03/13/making-it-virtually-e...
I should also say it's a bit heartening to know of other e-com groups out there benefiting from an in-house stack rather than attempting to go 'TO THE CLOUD!'.
One example where virtualization is easier in practice is where you're not in control of the network intfrastructure -- ie: you subcontract management of switches etc to someone else.
For someone the size of Etsy that wouldn't matter much -- they have an obvious need to manage the entire stack. But for smaller institutions that might not target "scaling up to serve the entire public web" -- it is a factor.
Another example is if you have heterogeneous hardware -- it is much easier to just install a hypervisor that runs on every box and then manage that via your vm-toolset, that set up many different images. It also makes it easier if you need to run different os' -- do all your os images have drivers for all your hardware?
Granted, in theory OS' should abstract away hardware -- in practice, if you need to change out a few of your mysql instances running GNU/Linux to Microsoft SQL servers for a new project/change in workload -- being able to just spin down the old vms and up the new ones helps a lot in terms of agility.
But yes, it's absolutely possible to provision physical hardware as well.
I bought mine (a Supermicro 2U Dual 3GHz Xeon 4GB 12 Bay Storage Server) for 400$ on eBay, with great service and terms. Recommended if you are ever in the need (or just want) a private server rack unit (1U, 2U or more).
1U with 2 CPUs, 10 x 2.5" drive bays, redundant power
http://www.supermicro.com/products/system/1U/1027/SYS-1027R-...
I have one bad thing to say...they are loud, really _loud_.
Several years ago I had a 1U SM server in a rack full of HP servers I could hear the supermicro over all off them.
The 4U pedestal I have at home as a file server is the "super-quiet" model with 8 3.5" bays in front and it is still insanely loud. I had to to rig up a fan controller just to keep it livable to have in the apartment as the motherboard couldn't slow the fans down enough.
One of the rows in our data center, lots of Supermicro goodness. We use the 12 bay 'jupiter' chassis for 2x SSD and 10X 2TB spinning rust per server.
It sounds like a single switch failure could take down a large amount of their infrastructure. The seeming lack of geographic (or even datacenter) redundancy also seems a little dangerous at this scale.
You're right that a single switch failure would be impactful, but the impact of one switch going down would actually be very minimal. For example our web servers are spread out between many switches. The switches and network equipment have greater redundancy between them. - Avleen Vig, Staff Operations Engineer, Etsy
I've seen situations where folks have done this because they believed they would need VSS/VPC or similar on the switch side, but ALB on the server should avoid that.
Perhaps they have a n array of access switches, each serving some subset to lessen the impact?
Or keep it simple. One nic for prod traffic, one for private/control plane. Separate TORs are nice, or just don't oversubscribe them. Keep critical data hot across racks. Plan for failure of any given host, then for dc failure. You'll have site power or transit failure waaay more often than a bad tor.
We've also invested in a 10gbe infrastructure rather early (we're smaller than etsy traffic-wise) and found it to be very useful for avoiding congestion issues when we need to transfer large files or pull data from a busy Postgres server.
As for HP servers, we avoid them since our DL360 D6 reported obscure fan failures when booting with an Intel 10gbe card and we had to buy the (slower) HP NetXen-based cards instead. Supermicro boxes are not that picky and you don't get the feeling that a vendor might be using dirty tricky to lock you in.
"In the future we may even run fiber to these machines" Anything against 10g copper? Per port prices are getting crazy cheap.
"DNS servers, LDAP servers, and our Hadoop Namenodes ... need RAID for extra data safety" Anything you've had issues with? Host level redundancy with the first two has been pretty great IME, can't comment on name odes. The only issue I've run in to is poor LDAP clients not fast failing an unhealthy server when another is available.
No reason against 10G copper specifically - we haven't had to address the problem in detail yet. When we do, we might choose copper. Depends what happens when it happens :-)
We had a very nasty incident a few months ago, where the drive in one of the LDAP servers died. Well, it sort-of died. It started to time out a lot but didn't go fully offline. openldap kept running, but when you connected to it, the TCP connection would open and hang. This meant that all of our servers saw the server as "OK", but LDAP stopped working and caused all kinds of brilliant havoc :-)
Sure. Premature optimization and all.
Huh. SSD from a major chip vendor perhaps? I had one of those bite me just a few weeks ago. in my case it would have been fine if the drive offlined itself. but the intermittent io pauses killed performance, but not health checks. Actually, same type of problem RO FSs make me scared of.
Probably time for a refresh!
- Avleen Vig, Staff Operations Engineer, Etsy