That sounds really good in theory, but in practice it's less good.
There are a number of different
types of costs to consider:
1. The cost of server hardware
2. The cost of unused hardware capacity
3. The administration cost (people, skills, etc)
4. Opportunity cost
The Cloud(tm) excels at addressing some of these. You don't have to pay /as/ highly for staff to manage you machines and network (the provider does much of that for you). You can run smaller, cheaper instances much closer to their limits.
The downside is that you pay a premium to the provider (even if you are your own provider). Additionally you lose a great deal of opportunity cost. If you use the "excess" capacity on the weekend for hadoop jobs, and need to stop them because you have a large burst of traffic, you've hurt yourself.
Your non-production hadoop jobs can also have unexpected and unintended impact on your production web servers, causing your users pain.
Given these things, if you focus is on keeping the best experience for users then you should split things apart.
The actual gain you get from cramming as much as you can onto one piece of hardware is much, much lower than you might expect. It ends up being easier just to get more hardware and dedicate that hardware for specific tasks.
- Avleen Vig, Staff Operations Engineer, Etsy