But often times those cost-benefit analysis don't take into account how quickly you can improve and work on your infrastructure. The performance or cost improvements need to be a lot to slow down your team even a bit.
Especially for startups this can hit them very hard.
I think engineering for 'The Cloud' can be just as time consuming if not more so for start-ups. For Amazon at least you have to engineer for failure. With a dedicated server configuration you still do need to engineer for that, but you aren't worrying about if depending on the solution providers block storage is going to take certain aspects of your architecture offline when it goes down (yet again), or if the portal won't allow you to bring up/down new instances during a very critical period of time, etc.
Generally speaking, most sysadmins at small places do a horrible job engineering for failure. Or, worse, they don't have professional SA's, just devs who know enough to be dangerous.
So while you don't run into a US-East EBS failure every now and again that affects millions of servers, you are subject to random unforseen failures because somebody didn't do X correctly. (Where X could include: disk configuration, HA configuration, DR plan, backup/restore plan, os management, firewall management, etc)
There's no one right answer here. I've rolled specific hardware solutions managed by a single team for certain applications, and everything was fine. I've been stuck working within the bounds of a managed services provider, and things worked too. Cloud is just another model.
EC2 lets you roll a globally distributed solution with good tooling for low cost.
This is the Crux. There are other options available, but going cloud makes it very easy to roll out something big without the necessity to have experts in all the lower level parts of your infrastructure. It lets you focus and the price for that is absolutely reasonable
1. If you're using tons of servers, you might benefit from hosting your own hardware on a large scale. But even then you can probably negotiate with the cloud service for a better price (like Netflix did). If you're a big customer and you think you should move off the cloud, the provider can beat your expected costs and still make a profit. Your team doesn't have thousands of years of combined hardware experience, and you're not buying hardware at a better price than Amazon got.
2) You don't want The NSA snooping on your company's data. This is a moot point because they will obtain the data anyway, with a gag order, if they really need to.
Most small companies aren't going to benefit from the elasticity that the cloud provides. The administrative overhead will be similar on each except in the case of the cloud you need to worry about the persistence of data either via EBS, replication or having an acceptable loss level.
The question is whether or not you're paying up-front for hardware, and managing it locally somewhere. If you lease a $10/month VPS, you're using cloud services.