Hacker News Reader

Comment by forgot-my-pw | Hacker News Reader

I would think if a business use more than 500 servers, it would make sense to start considering having your own datacenter?

View on HN

It’s not either/or. You want to be in a position where your own racks and public cloud providers are all running instances of the same distributed scheduler. Then you can buy capacity wherever it makes the most economic sense at the moment. Just proving that you have the capability to run your own hardware cost-effectively can negotiate Amazon/Google down to that price or lower.

This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.

I wouldn't do my own datacenter as in build out a building, but I would certainly lease a cage and build out my own infrastructure at probably half that count.

As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means.

For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.

I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article.

With Netflix's Open Connect (https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances.

Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc."

At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.

I've used most of Hashicorp's tooling - Consul, Nomad, Vault, etc. - and I consider myself to be a Hashicorp fanboy. But if you're spinning up a bunch of VMs "in the cloud" and that's all you're doing, you're not taking advantage of the platform. The idea behind using AWS is not just to avoid your own capital infrastructure, it's also to allow someone else to do the "undifferentiated heavy lifting". In our case:

Consul -> ParameterStore

Vault -> KMS/Cognito/IAM

Nomad -> Elastic Container Services/Fargate (serverless Docker)/Lambda

SQL Server -> RDS -> Aurora

Memcached -> ElastiCache (Memcached protocol)

Consul+Fabio (service discovery/load balancing) -> Elastic Load Balancers/Route 53/Autoscaling

VSTS (MS's hosted CI/CD platform) -> AWS Code Build/Code Deploy/Code Pipeline

"not marrying yourself to a one cloud provider" is equivalent to "coding using a repository pattern to access the database" in both cases people claim to not wanting to tie themselves to one vendor and then they realize that they aren't taking advantage of what the vendor has to offer.

In my experience, no one wants to go through the pain of moving to another vendor just to save a little money.

That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do.

There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.

I would never roll my own unless I was of facebook/google size. Colo space is abundant and affordable.

Netflix doesn't pay retail for AWS or G Cloud

This isn’t such an obvious decision. Running your own datacenter is difficult. Besides racking and stacking, you need to deal with power and cooling. Most likely you will outsource all of this to a collocations management services initially, which becomes your overhead just like AWS/Google. But they have economies of scale, in addition, they have expertise in managing all of this equipment with very high reliability. They also have tons of spare capacity (elastic) that are at your disposal, you’ll lose this in your own datacenter.

On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms.

At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen.

Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).

The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC

In general, I think it’s a wash. The margins you save on end up costing you in other ways, such as development velocity and having access to easily deployed high-scale solutions like dynamo and managed instances like RDS. it takes time and organizational know-how to get to a point where similar things can be managed in the same way as AWS.

Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.

The margins you're paying for include functionality you don't use and your own cost of integration, observation, and management of their stuff. The principal advantage of using a cloud provider is avoiding capital costs and any useful apis they have that you'd otherwise have to develop/maintain in-house.

The most fun thing about startup world is that it pretends that normal rules of economics and accounting do not apply to it (until a startup runs out of money)

$X in CapEx makes balance sheet look better than $X in OpEx.

As generously as I can put it: It's a form of denial that people use when overwhelmed with everything else. There's typically too little real data on what the money's going to look like until a bunch of things are tried out, each essentially a random variable. So instead of freaking out about that, just bandage over it with some lies to yourself until you can actually start addressing it.

Businesses in well-known markets can start estimating their costs + potential revenue early, and optimize early. But for a business that might "pivot" a few times before finding "fit", how do you do it? You just look at time (runway: burn rate * remaining capital) you have to try stuff.

It's not great, but I can understand why some people do it.

If you need to pivot before you are cashflow positive, then you dont have a business and you never had a business.

I keep being amazed that total business failures that result in pivots are considered to be successes in the startup world.

There are lots of open source products that offer programmatic/desired state/idempotent config & apis for provisioning and hardware lifecycle management in private dc.

There are not even close to how good and mature the AWS tools are.

You really have to factor in the labor costs of infrastructure and operations into that as well. Running your own private "cloud" isn't cheap. And no matter what you do, you aren't going to get even close to feature parity with the public cloud offerings (or even speed, if you are using virtual-machines). There are a lot of trade offs that you have to weigh in that don't always come down to bare dollars and cents.

Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so.

I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.

If you're a margin business then certainly. If you're not then there's a lot to be said for keeping your core competency lean and offloading the heavy lifting at a negotiated price.

This is assuming you can do everything at the same efficiency level as Amazon or Google. It's likely that you wouldn't be able to outperform on these metrics to a degree that it would make the profit margin difference a wash.

The exact point I am making is that at a certain point, you are able to do things as efficiently as they are. Netflix certainly qualifies at that scale.

It's different for everyone.

If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.

Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.

I completely agree. I jokingly tell everyone I know using AWS to "not use us-east1" and that they need to use another region not just AZs...and they always laugh and say "you were right" with us-east goes down.

At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.

How often does us-east-1 go down? I know of about 4 big outages in the last 7 years but you're making it sound like it happens much more frequently. Is there a list of outages somewhere?

it's not just about it going down.. if you rely only on us-east they have had multiple times when you couldn't autoscale because there was no more capacity... if you rely on running lean and scale/up/scale down events us-east would have bitten your ass a lot the past year... Just for comparison sake even our private colo has had 0 issues the past decade that caused any interruptions to service

I do run such a service, one that must scale daily in response to traffic at multiples of off peak traffic and I can state unequivocally that this is false.

I can vaguely recall over the past several years single availability zones having short periods of time where single instance types were unavailable, but never the entire region... The idea that this happens "a lot in the past year" for the entire region isn't my experience at all.

It's a good question, but depends on your workload.

Netflix is all about content delivery -- which is both distributed and highly variable over the day/year.

Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.

Reply on news.ycombinator.com