Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

Having done this computation a number of times, with these machines it would flip over to 'colo' at about 150 machines. The really interesting thing is that amazon costs scale linearly with size while owned infrastructure scale fractionally. So the advantage just keeps growing and growing.

Bottom line, if you can host your app and resources on less than a 150 machines in Amazon you win, more than that and you're leaving money on the table. Once you get to the point where you are deploying new datacenters to support your customers you get a huge boost in operational efficiency.

View on HN

Still not convinced about this. I can get a pretty beefy server from Hetzner for 60EUR per month. Skylake 6700, 64 GB RAM, 500 GB Raid1 SSD. One time setup fee of about EUR 100 but can be canceled on months end. On AWS an equally specced machine would run >300 USD per months easily. So it really only makes sense if you can scale your usage up and down a lot during the day. Does not make sense for most of smaller projects. Most of the companies i worked with had an EC2 setup of 3-10 instances that were mostly always up, which is way more expensive than doing the same thing on Hetzner. Basically they just did it because of free AWS credits for startups.

Slight digression here but "RAID 1 SSD" might not be a good idea, especially if you bought two SSDs at the same time. They are probably from the same lot. My experience with 3,000 SSDs in use is that they seem to fail with a combination of age and rewrites. So I would be concerned that two equivalently aged SSDs with an equivalent set of rewrites (as RAID 1 does) would be quite likely to fail at the same time, or more precisely, when one failed you probably have a statistically harder time recovering the mirror from the other.

That is just statistics of course, but one of the things I tried to do in the Blekko infra-structure was to mix the ages of the SSDs to mitigate this risk.

This is good advice even for spinning disks. I once met an unlucky RAID-5 array of three identical disks from the same brand, model, batch and with close serial numbers. When one of them failed, I immediately ordered 3 replacements.

They didn't arrive on time.

Good thing I had backups.

I've heard there's strong correlation in failures by lot number. Can you confirm this?

What's the practical way to avoid this? Staggering your SSD purchases?

This is a false dichotomy. There's no need to flip to colo, when dedicated hosting exists. The provider takes care of the network, hardware, and potentially much more depending on the service agreement. You can even have the provider setup a cloud environment for you, if you want to be able to spin up and down VMs. This eliminates the capex costs, while having much lower opex costs than colo at smaller scale.

Considering that you can easily do dedicated for 1/7th the cost of Amazon for the equivalent amount of resources, it doesn't take anywhere near 150 machines to realize cost savings, it's more on the order of one complete physical machine as a dedicated server. You can purchase a 2nd one for redundancy and/or for always available additional capacity, so that you can burst to double your usual peak performance and still save 5/7th the costs of Amazon. If you're expecting to require the ability to scale much higher than 2x, you can also purchase a bunch of servers just for 1 month, or even less time depending on the provider, or take a hybrid approach if you really need an extreme amount of dynamic scalability. The reality though is that such extreme scaling is in the minority of cases.

I'm not sure how you arrived at the 150 machines number, but that sounds like 3-4 full cabinets and about right to have a reasonable scale for colo.

Note: I manage dedicated-server hosting and all web services on Titanfall @ Respawn.

Surely your calculation doesn't / can't include bandwidth charges? And if you have Windows hosts... cloud screws you too.

You're not wrong that it takes a lot more machines than most shops need, and if you can make your scale elastic, 150 machines probably goes a long way! And to your point, developers undervalue their time a lot, and dealing with colocation and sourcing of hardware and backups can be a real drain to save some on CapEx.

Our transit costs out of our Colo in Santa Clara are $6K/month for two dedicated gigabit lines. That translates roughly to 518 TB of data transfer a month if we could keep the pipes fully lit 24/7. Cogent has offered to make one of our gigabit lines burstable to 10G if we would like, same cost if we don't exceed a monthly average use of 1000Mbps.

And we use Linux so no Windows hosts charges.

There are more and more tools that are force multipliers for your Site Reliability Engineer (SRE) equivalents. You can't avoid swapping out dead drives, but with the right systems architecture you can make it pretty painless.

Wow, you're getting ripped off.

Indeed, I've seen Cogent go for $800 - $1000 for a 1GE over 10GE commit.

It's possible to only pay the lower end of that range for a 2Gb 90th percentile commit on a 10Gb connection out of most major markets. Cogent pricing fluctuates wildly depending on your ability to negotiate.

Ignore the whole song and dance when they have to bring in a VP into the call to specially approve the pricing they're trying to pitch to you at. Those are standard transit sales tactics. Or avoid Cogent's sales entirely by working with a reseller.

Speaking with the VP to Negotiate for transit prices? Sounds like a pita. AWS knows that you're an engineer who just wants to write code and has made that as frictionless as possible.

This goes back to my first point in that going into the DC requires staff with very different skill sets. It's not that it's not worth it, but there's a lot of costs involved part from the per-hour instance cost.

Can you elaborate? Discussions with details about things like this are always interesting.

He's paying 2-3x market (although 2Gbps isn't huge). Cogent is pretty much bargain-basement, too.

I wish the industry would stop quoting prices they way we do (USD/Mbps, so e.g. $1.20/Mbps); Gbps is probably the right unit now, and it may make more sense to invert.

But there are lots of terms involved in a transit contract; unless you're buying tens of gigs, crossconnect, port cost, commit, etc. may be more meaningful than per-Mbps cost. And pricing often depends greatly on exactly where in a city you're buying (at IX, carrier neutral facility, etc. will be cheapest; off net building or monopoly building will be most expensive) -- and of course Cogent transit vs., say, Level(3) transit are only superficially the same.

Exactly, transit cost is the big killer.

AWS/GCE/Azure are all really expensive compared to even good multi-homed transit. If you push more than 10TB and you don't need multi-homing you are already ahead.

If you do need HA multi-homed transit you are looking more at 50-100TB or so but still, there are a lot of places that easily do that every month, especially if you are doing cross-DC storage replication for example.

Yes, but a huge reason to go with Amazon is to be able to take money back off the table. Sure, if you have predictably 24/7 load, then you may be better off racking your own at 150 machines. BUT, if you're not sure how much you'll need, and your app is built to scale up and down with demand, then that 150 number starts to slide upwards.

Risk mitigation is part of the cost of infrastructure that needs to be factored in, and if your business has a downturn, or your compute needs turn out to be lower, then you can scale down your OPEX pretty quickly. If you had bought them, you'd be stuck with them. (See Zynga)

The operating cost of doing it yourself is always hard to pin down, and include many many things. OS/Hypervisor licenses, staff, hardware, cooling, redundancy, disaster recovery, power, or Colo fees etc. The advantage with AWS, is that this just a few numbers on a few various services and it is very predictable.

Yes, I guess the 150 machine is more like a guideline. It depends very much on your workload. If you need 150 beefy machine running at 90%CPU all the time, then yes, running your colo may come out ahead. On the other hand, if you run a system with an online fleet of say 20 web servers, a couple of beefy databases, and some offline analytic workload, then AWS will definitely come out ahead.

And that is not considering other AWS offering. I've come to really like DynamoDB. It takes some time to get used to, but I've found it's solving more and more problems for me, without having to scale OPS engineers. There is a danger of lock-in, but I guess as I am a paying customer, it's not going away anytime soon, and Amazon is probably not jacking up the price either.

Yet Netflix migrated to Amazon and I think that it has way more than 150 machines.

Yes they did. And that is a really great exercise in thinking about what their profitability would be like if they were structured more like YouTube in terms of their server infrastructure.

But to the point of many other folks here, if you're all cloud you can really shrink fast if you need too. Lose a millions customers? No problem, just drop some machines. Any business that can price into the monthly customer fee the marginal monthly cost of AWS to support that customer, can keep their costs flat when their customer counts vary. Someone with dedicated hardware is still paying the bills for it, even when their customers leave.

I am not sure I'd agree that Netflix could've been more profitable if they went with their own DC. I think Netflix does not utilize that much hardware. Their streaming is on their own CDN, which I heard that are placed directly in the big cable companies' DC. Tracking and storing user profiles and other meta data would not require a huge amount of hardware. I think Netflix also does quite a bit of video encoding and analytics, but these workloads would fit well with EC2.

That would be the thing right? Clearly they can compute their marginal cost per subscriber, and they know what the lifetime value of a subscriber is, so they budget what to spend on EC2 infrastructure and movies at Sundance and still meet their net income goals. They would also know how much difference it would make if they needed to add or reduce their EC2 footprint.

Sadly I don't expect them to share what those costs are. I have always felt that Amazon could, in theory, always out compete with NetFlix with their Amazon Prime Video because they would only have to charge the marginal rate for the hardware. However to date it seems like the content contracts have kept NetFlix on top.

I have to think they're getting a deal.

Plus, they're the number one poster-child for AWS. Netflix running on AWS makes AWS safe for CTOs everywhere. So, yeah, they're definitely getting a deal.

Which should be expected when buying in bulk.

that would be fun to calculate

Bulk rates are always going to be unlisted and cheaper.

Also, having 150 machines in a single location is rarely optimal.

Reply on news.ycombinator.com