Comment by NathanKP | Hacker News Reader

Comment by NathanKP | Hacker News Reader

Parent Full thread

> You just have to have the discipline to run your internal data center the way cloud providers do.

AWS employee here, so I am obviously cloud provider biased, but in my viewpoint having that discipline only comes from having very good, expensive engineers. It's hard to get people who really know how to run a datacenter well, especially when most of the good ones would probably prefer to work for a larger cloud provider since than is more interesting and impactful than running a smaller datacenter for a single company.

The forgotten / underestimated cost is the salaries of these people to set things up well in the first place, be on call to fix things when they break, and perform those upgrades in the same way that cloud providers do.

A single engineer salary can get you a LOT of cloud compute power on AWS. If a cloud provider allows you to build the same thing with two fewer engineers then that's probably an extra $400k or so for the infrastructure budget after you factor in these engineer's salaries, taxes, benefits, etc. And $400k can get you an incredible amount of AWS cloud compute.

> The forgotten / underestimated cost is the salaries of these people to set things up well in the first place, be on call to fix things when they break, and perform those upgrades in the same way that cloud providers do.

The thing is, you need this equivalently for any cloud provider as well. People who know how to operate and run a cloud account effectively are not a dime a dozen, and given the migration going on now are actually in higher demand than some qualified datacenter operators.

IMO the best examæel is spinning up a DB with nightly snapshots. This is trivial in AWS. Doing it in your datacenter takes planning, decispline, skills and time.

Same goes for backups to S3 vs. setting up a SAN on the datacenter.

Managing AWS infrastructure isn't trivial, it's easy to forget what that instance does, if everything was provisioned with point and click. Cloudformation or terraform makes it doable. But technical debt is also something you can defer payment on :)

Yeah the DB example is particularly relevant. Amazon Aurora now has continuous point in time backups so in the event of something going wrong you can roll the database back to an arbitrary second: https://aws.amazon.com/blogs/aws/amazon-aurora-backtrack-tur...

Additionally Aurora has robust zero downtime self recovery from a variety of error conditions including disk failures, and you can test that out by running a SQL query that simulates disk failures for example: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Auror...

This level of operational excellence is incredibly hard to build yourself either on premise or even in the cloud if you are only using VM's. Time and time again you see companies that thought they had backups but then discover the backups don't work right. It's not because these companies are stupid, its because it is hard to do right, and building and testing this requires a lot of engineer time. When you do take the time to hire the right people who can do it right you end up finding it is way more expensive to do it yourself than using a cloud managed service. AWS is able to offer the service much cheaper because the cost of development and maintenance, etc is spread across a huge number of customers.

One thing I've noticed with cloud vendors is you are inevitably a small fish in a big pond. Few companies have the AWS investment to make Amazon take notice. Essentially, if something gets screwed up you have no recourse beyond a refund and moving your infrastructure.

If you have encountered a specific issue where you feel that Amazon ignored you because you weren't a big account please email me using the address on my HN profile. In my experience both on the customer side and now the employee side we are very fair to customers of all sizes when it comes to issues that are our fault. We do have a shared responsibility model though, so if a customer makes a mistake like terminating an instance that had important data on an ephemeral EBS volume, or similar there isn't much support can do.

My comment is more of a generalized observation. Your response seems to confirm it. Who makes the call on "shared" responsibility? I also guessing there is some sort of arbitration clause. (found it: https://aws.amazon.com/agreement/)

I'm not saying any of this is necessarily wrong, just not what people often expect when they dump, to them, a significant amount of money into something. Many midsize and larger companies like to "big dog" their vendors. They are not accustomed to the cog in a machine feel that most of us tolerate daily.

Reply on news.ycombinator.com