Hacker News Reader

Comment by mikeklaas | Hacker News Reader

A couple of problems with this analysis:

Using reserved instances would push the EC2 figure down 30-40% (and like the dedicated option, provide further cost savings in years 2 and 3)

You can't assume that the marginal increased technical management cost is zero. If that's true, you're employing people that aren't doing anything productive with their time. A dedicated cluster of this size would likely consume 50-100% of one employee's time, which adds at least another $50k to that side of the ledger.

View on HN

Going dedicated has not increased the workload for us over the cloud. If anything, it has reduced it significantly.

(We have over 150 servers)

DanBlake I share your same sentiment. We went the real servers route and when someone does if they use programmatic methodologies for system administration (Puppet, Chef) you can have systems online just as quickly or systems reprovisioned in to different roles very quickly. We call it our own private cloud!

Edit: Note we are using an automated Debian preseed with Puppet installing from an internal repository to bring the systems online with minimal sysadmin interaction.

Is there some place I could read more about how something like that works?

Yes, but if you have a cloud cluster of this size, it would still require similar time for systems administration, not? The only real difference here are hardware failures and replacements, but for a cluster of this size, I do not think that enough hardware failures will occur to occupy a technician that much compared to the cloud.

No, because administering the hardware is additional work, a burden that a commodity (cloud instance) does not carry.

I think this is a false economy. You're just paying your provider to employ people that are capable of fixing these problems instead of in-housing them.

By and large, if you manage physical, owned infrastructure the same way you manage cloud resources. That is: automated configuration, monitoring, and deployment. The vast majority of the time it's the same process. You just need to make sure that your instrumentation is sufficient to notice when things aren't running correctly and remove machines from service until someone can figure out what happened. This could be in the form of an outside tech or in house support. Either way, at clusters of this size a loss of one, or even 5 machines should not be the difference between everything working and a colossal systems failure.

I don't think it's safe to assume that paying for that service being done by someone maintaining 1000+ machines is necessarily going to be more expensive than paying to do it in-house for 10 or 100. It's generally going to be significantly more cost-efficient on a per-machine basis to have someone administrating 1000 machines vs. 100. If nothing else, the cost of specifying and purchasing hardware and getting staff up to speed on it is going to be spread much more efficiently, as is the cost of 24x7 oversight and backup hardware. Not to mention they're probably buying all that hardware significantly cheaper.

I think we are agreeing here, we're just phrasing it differently. I didn't mean to imply there was a direct cost comparison; just that part of your payment is for them to fund someone's paycheck. In the grand scheme of things it's definitely cheaper to maintain a high ratio of machines-under-maintenance to systems administrators.

That being said, the difference is Amazon pays an administrator to watch potentially thousands of machines of which you may be using a tiny fraction. You're essentially paying for a very tiny fraction of an admin's time (somewhere) who isn't vested in your business nor does he care that your site is acting wonky. He's just answering and working through tickets (if you bought support). Cloud providers can be come markedly less helpful the instant you (attempt to) deviate from the very carefully maintained cookie-cutter pattern they've set out for you.

Working at a company with hundreds of machines and dedicated admins for them, I can tell you that unless the problem has immediate revenue impact, they're not going to handle it a whole lot differently from a cloud provider - you have to cut a ticket, it's prioritized, and it goes in the queue.

On the flip side the $100k cost for buying dedicated equipment is a first-year cost only, and the equipment can last up to 10 years, where at AWS $350k/year is every year.

Reply on news.ycombinator.com