Hacker News Reader

Hacker News

Comment by andrewstuart | Hacker News Reader

Parent Full thread

andrewstuart

Seems crazy to run such large infrastructure on a major cloud.

Tens of millions wasted.

Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible.

S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave.

S3 is 9 cents per GB egress fees.

Cloudflare R2 charges zero egress fees.

At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong.

View on HN

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of spend you'd be crazy to pay the full $0.09/GB.

Edit: Note also that Snowball egress is $0.03/GB. Slightly higher egress, much lower setup cost. You'll have to do the math but they're both clearly attractive options vs. full price $0.09/GB egress.

Cloudflare R2 also has a tool in beta that provides an incremental migration from S3. As users request files using your R2 URLs, Cloudflare automatically migrates the files from S3 to R2 on request. Depending on your user's request patterns this may be a way to migrate everything without paying any additional cost. If you have a lot of files that are rarely accessed then they will be slow to migrate, but because they are rarely accessed your S3 costs could be significantly reduced by using different storage classes anyway. Eventually you will have a small enough number of infrequently accessed files remaining in S3 that the additional cost to migrate them all in one go can make financial sense.

Awesome. I just wrote about it above. Thanks for sharing!

I worked at a place that did an AWS to GCP migration. I didn't look into the details, but they used a provider that basically gave them a fat, dedicated fiber link between the two (they picked the same GCP region). It was good enough that requests could have data cross clouds. It was expensive to set up, but it saved on egress costs and opened up options for migrating services.

I believe the relevant GCP product is "Dedicated Interconnect" and for Azure it's "ExpressRoute Direct" but don't quote me on that; I only know AWS. In my example you can swap out the Internet port for one of those cloud interconnects and go straight from AWS to your new cloud. That's certainly what that provider was doing. I bet there's lots of good money in facilitating those migrations using this relatively simple technique.

Plot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)

There's this law on the internet or SO or whatever where if you put the wrong answer in a post.

Someone else in the comments will correct you with the correct answer.

This in effect.

This isn't just the internet. I've used this at work several times to break stalemates. I've come into many stalled projects where people are arguing over trivial details or blue skying on requirements. 1-2 hours and a 7 page design doc later I'll have everyone review what I know is an 80% answer. Pretty much every time they'll all start attacking my design, pointing out minor issues and then I have them. They've agreed to my overall design and are into details.

Same if someone is calling out all of the general problems with solving a problem but not providing answers where you can't get them to bite.

https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham'...

Cunningham is cunning!!!

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1), the price of building your own services, the price of software you need to buy, and so on.

You only save money if the differential cost based on all the factors is lower. AWS is expensive for some things so it often does save some money, but not always, and if you haven't done a proper analysis you can't know.

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops

This is the traditional “sell” for cloud computing and I don’t buy it at all.

The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems.

The clouds sales pitch is the if you go cloud then you don’t need all these specialists.

That’s rubbish. Cloud operations need the same or more headcount if technical specialists, they’re just doing different things.

The old “don’t run your own systems, it’s cheaper and easier to go cloud” is just sales fiction.

Don’t believe it.

Don’t believe it.

You don't need to 'believe'. You need to do an analysis of what you need and how much each option will cost. Then you can know.

If your argument is "I believe onprem saves money" or "I bet AWS is cheaper" or "Jim Morrison came to me in a dream and said I should use Azure" then you haven't done enough research.

> you haven't done enough research.

the research isn't free either. The more indepth and time it takes to do such research, the slower you come to a decision and ship.

Cloud allows you to ship fast. It allows you to go without research - just accept the marketing, and pay up.

You pay above-cost (compared to on-prem) when you grow to a certain size. But this is usually a worthy trade off tbh.

It's a good way to bypass legacy IT teams. A onprem server will have a bunch of snake oil endpoint protection products running on the box, bespoke config changes and take a couple months to get up and running, a ECS container is up and running in minutes and has whatever you shipped in the container without all the commentary from the peanut gallery.

A poor DX can happen in the cloud as well. Waiting weeks for IAM configuration to be solved or a security group to be opened..

I've run into this kind of snake oil in the cloud, as well. Some organizations demand all routing go through an "upstream" VPC operated by the parent org's IT dept, so various third-party security services (WAFs, IDSes, etc.) can scan / inspect the traffic.

When I moved reddit from on-prem to the cloud, I cut our costs by 27%. It was the same number of people managing both infrastructures, but once we moved to the cloud, I no longer spent most of my time imaging machines and driving them to a datacenter to rack and stack them. Instead I spent my time coding ways to manage machines via API, so that when I needed to double our infrastructure, I just ran a script.

Cloud makes a ton of sense for high growth companies. If your infrastructure is mostly static, that's when it makes sense to go to a datacenter. Or if you have one very specific use case, like Dropbox.

This is very unusual in my experience, except in cases where the on-prem was either massively over-provisioned or the customer was getting absolutely gouged on pricing at the DC or IP transit or whatever.

Do you recall what the big savings were in? Compute, storage, DB etc? Be really interested to understand why this case is so different to many I've read.

Funny enough I have the exact numbers handy! Keep in mind this was 2008 and we only had 150M page views a month back then.

Data Center (per month)

Servers: $6K

Cabinet (x3): $15K

Bandwidth: $2.5K

Support: N/A

Total: $23.5K

EC2 (per month)

Servers: $13K

Storage: $1.5K

Bandwidth: $1.1K

Support: $1.2K

Total: $16.8K

Interesting, thanks - I assume on EC2 you were running your own database servers? (Just based on the fact that you don't mention RDS.) This would keep costs down significantly in most cases although I assume they get transferred into staff costs if you've got people maintaining them to get the same level of support for things like backup, replication, etc that you get for "free" in RDS...?

Yes, we ran Postgres on EC2. I admined it myself both in AWS and in the datacenter. RDS didn't exist at the time. :)

At the time we and Heroku were running the two largest Postgres clusters on EC2.

There are hundreds of factors for each company and application. Sometimes it works and sometimes it doesn’t, but it’s rarely as simple as retail prices.

Even the same number of "technical specialists" doesn't mean it's the same if one option lets you move faster or remain more reliable.

This is precisely my experience and what I observe in companies that fall into the cloud trap. In addition, it is important not to overlook the significant transformation and upskilling process required, as well as the time needed to accomplish it.

Datacenters are expensive to run, for many reasons. That is the appeal of the Cloud right there, before we even start talking about the people aspect.

I challenge any in-house team to achieve the same reliability as S3.

To be fair most businesses probably don't need it, but it is worth taking into account.

the same reliability at what scale? this is a major difference. S3 has amazing reliability but is a incredible complex system with many moving parts, which incurs cost in terms of reliability. The simpler the system, the more reliable it usually is. Most bussiness do not operate at even remotely the same scale as amazon, and running a small scale cluster of S3 like functionality wouldn't be that hard.

S3 is reliable as data isn't not stored in a single place. It reduces the risk of water leakage or some vandals ripping your storage rack apart.

However, as someone put it, put your money where your mouth is. Become a contractor offering to reduce storage costs in exchange for 10% of the savings. You should be a millionaire in no time, according to your narrative.

Amazon S3 standard storage offers the following features: Backed with the Amazon S3 Service Level Agreement. Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.

> That’s rubbish. Cloud operations need the same or more headcount if technical specialists, they’re just doing different things.

It's a capex vs opex question. For financial quackery reasons, accountants and stock markets prefer having less people on the direct payroll, which is why everything not part of the "core business" is outsourced - even if it is more expensive in either short or long term.

> And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1)

This is an important factor to remember when evaluating could costs. If you want to survive a cloud outage, you need a multi-AZ or multi-region deployment, and that costs developer hours. And you need to deal with the potentially catastrophic cost of inter-AZ or inter-region traffic, which can be catastrophic and/or cost more developer hours to mitigate.

> If you want to survive a cloud outage, you need a multi-AZ or multi-region deployment, and that costs developer hours.

RDS databases are one click to set up in multi-AZ, and if your stack is Kubernetes based or at least EC2 autoscale capable it isn't much more work to make it multi-AZ as well.

Multi-Region deployments however, these are indeed expensive and nasty to set up.

usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.

^^^

The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where they deal with most hardware, etc.

That said, maybe Canva has considered a non-AWS solution and decided against it. Or maybe they've gotten certain better deals from AWS. We can't really know for sure on the outside.

My team uses AWS for _everything_. There's three reasons for this on our team (and I've researched it for our use cases.

- Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge.

- Engineering Costs. Similar to above, but in terms of engineering, there's less to know and understand. Instead of needing to know how the AWS load balancer routes/connects to a VM somewhere else, and how that VM gets it's blob-storage-data from azure, we only need to understand AWS concepts.

- Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being.

Now, we're a 50 person company so YMMV, but the above tradeoffs plus an "opinionated" setup in AWS (everything on ECS, logging to Cloudwatch, RDS for DB) drastically reduced the "ops" overhead on our side after the initial setup. If I started over, I'd make the same decisions again.

> - Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being.

This is where I think the FCC should take action.

To the extent that this issue is a mutually agreeable arrangement between you and Amazon, it seems obnoxious but does not seem like it rises to the level where regulators should take action. But it affects third parties too: specifically, it prevents non-AWS-hosted vendors from effectively marketing their services to you. In that regard, I think the FTC should try to put a stop to this. AWS should not be permitted to effectively subsidize its and its partners’ services over outside competitors.

(And the US Government should never have accepted cloud deals with excessive egress costs. Part of the bidding process should have been a requirement for networking outside the winning provider to be priced competitively with internal networking)

But you're missing the point above. S3 is probably the easiest service to replace - there are loads of providers which use the _exact_ same protocol as S3. It's a drop in replacement. It literally uses AWS concepts, there is nothing else to learn apart from putting a different url into your application.

Very few people should really be using S3 at any serious scale is my thoughts. The cost savings are enormous (plus cloudflare for example replicates your data a lot closer to users for no extra cost, significantly improving performance). The cost savings can be absolutely enormous for very little/no additional complexity given how many providers are compatible with S3, and the fairly 'boring' nature of S3 compared to other technologies.

People really have to be afraid of running VPS or whatever if they can't spun up a min.io instance to have their own S3 without dealing with Amazon's Bullshit

How big of a production deployment have you ran on min.io in terms of data transfer / month and total storage?

Because I've done small scale and can tell you I'd run S3 in the future.

Point taken as I haven't used it in prod properly, I'll cross that bridge when I find it but Id rather keep my money and put some time into making it work. There's also cloudflare R2 now if S3 is too expensive but self-hosted is out of the question

> My team uses AWS for _everything_...If I started over, I'd make the same decisions again.

Okay? I never said that sticking to a single cloud provider isn't appropriate for some (or maybe even most) people. It's good that you have a setup that you believe works well for you.

or just consider different providers

and easily lower costs by 50% depending on balance storage/bandwidth

Nitpick: It's "you can _check out_ anytime you want but you can never leave"; the dissociation between checking out and actually leaving is what makes the line. Still a good reference for the S3 situation!

Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!).

We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of time where the expected load was low to maximize server utilization in those periods.

Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.

The term "private cloud" has existed for easily 15 years, pretty much straight after Google coined the term "cloud computing". The current deifnition of "on-prem" actually changed after cloud computing reached mass adoption. Prior to then, on-prem referred to "in your office" vs. "in a datacenter". These days "on-prem" means "not in public cloud".

Yes, it's basically the same as "on prem", but instead just a bunch of individual (virtual) servers they are managed by a system that provides typical cloud APIs, such as OpenStack.

It will mean compute resources that can be transparently shared amongst different things (which is where the K8s comes in) as opposed to boxes dedicated to a single service.

on-prem services but running with the tooling that is typically used for cloud computing rather than bare-metal on-prem of old.

Why do they need to get everything out? Some not-insignificant percentage of the content is surely archived/unused/unneeded assets. From customers who will never log in again or projects which are long over. Export the most recent 10% and put anything new on <new storage>

Absolutely agree. There are more nuanced strategies to migrate off a cloud provider. Could even enable a soft export for idle accounts, leave it where it lie and move it when customer re-engages.

I would think you could set up a migration process where existing things are loaded off of S3, and as soon as you make any changes, you save it to $NEW_STORAGE and delete it off of S3. Eliminates the need to egress anything, lowers your S3 bill over time, and as a user it'd be pretty hard to log in after a year and be upset that your FB cover image is no longer there, especially on the free tier.

Many aws services are hit or miss but s3 is just a marvel. Running geo-distributed 100s PB Ceph cluster with tiered storage isn't exactly trivial.

Millions of dollars per year isn't exactly a trivial amount of money either… Nobody said it's some super easy thing one could do on their free time, but given the scale it makes sense to build the knowledge in-house instead of getting vendor-locked-in at the most expensive service provider on the market.

Id argue this isn’t true until your spend surpasses $100MM. Lower amounts don’t justify the extra headache of on-prem.

I know we're just arguing about amounts at this point but that's pushing $2M a week. It seems on on-prem would make financial sense long before that.

Depends on your requirements. If you need 11 nines durability S3 is probably still cheaper

There's also the hybrids. Only storing "unlikely to be accessed" data on S3 would likely remove a huge part of the AWS bill, while significantly limiting the size of the on-prem data set.

Of all the possible related problems, object storage is the easiest.

I really hope that Canva's executive dont believe their product need such garantees…

> At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong.

If you're doing a one-time export you can use a Snowball, which would cost about $70K for 230PB, assuming you had somewhere to load it off to once you got the data so you could give the device back.

It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.

> It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful.

See the Docker x Cloudflare case study where improving cache-hit ratio by 2% (by moving to R2) decreased S3 egress fee by 66%: https://www.cloudflare.com/en/case-studies/docker/

I'd go for a hybrid approach: new data is stored on private cloud, and old data would be gradually migrated or phased out.

Most businesses treat cloud like a utility, not tech. Don't want to generate their own power, deal with sewage ... and compute.

Which in 99.9% of the cases is the right decision. Sorry for all you infra techies, but that's the way tech matures.

Infra techies are fine. Likewise 'grid is mature' and yet most of buildings with floor space over NN m2 have team of electricians, being them direct hire or contractors. Same for the 'cloud' - 90%+ of the world deployed 'compute' is not cloud.

Also, infrastructure fundementals do not change, now matter where they are running.

As long as dev's don't seem to understand real world limitations, (for instance, a network cannot be instantanious, and latency is not consistent) and thus leave tons of performance on the table, i think the "infra guys" will be just fine.

Also, don't forget there is an entire world out there of people making sure connections to the cloud are even possible across the internet.

Work for a 30bn market cap company, all our stuff is built on AWS. Offices all around the globe. Not a single electrician on payroll.

YMMV.

> Not a single electrician on payroll.

Directly, yes maybe. Many companies don't have in-house experts (and get shafted by vendors as a result, as they're lacking the competence required to assess the quality of work).

But indirectly, they all have. In Germany, at least, you're required to have electrical appliances inspected by a licensed professional (an electrician or otherwise qualified person) at least every two years - most companies opt for using dedicated external companies, but you can also train someone for that task. On top of that come all the electricians hired by the landlords - the ones dealing with complaints and remodels, or servicing lifts, escalators, HVAC, datacenters, ...

Utility is just an old term for "as a service".

Electricity as a Service, which includes payments for expert maintenance.

Thread is about AWS and merits of in-sourcing, which is not how the world operates.

Anyways.

You may call your company arrangement `electricityless` even! Like `serverless` - we don't use servers.

If you still make a profit and grow with public Cloud, and you had to risk time-to-marker, reliability, man-years, to make something on your own, then it's not that crazy.

More of a "good problem to have" (and you can solve it when you need to).

The real story isn't how expensive it is to leave, it's how easy AWS makes it to spend. It used to be that you'd see how many servers a project needs and have to beg someone for them, and then only get half of what you wanted. With AWS, it's much easier to launch something new, but there isn't the same amount of scrutiny on spend.

“Nobody ever got fired for choosing I̵B̶M̶ Amazon”

Cloudflare could build a product that not only caches, but transfers S3 to R2 on usage.

- Since non transactional data is likely to be on GitHub or somewhere, it should be easy to re-hydrate that on R2.

- Once enough time passes, there could be a sunset on unused data, or a strategy planned to 1-time exit write-off for S3 with those assets not on R2.

Standard migration procedure.

What happens when cloudflare changes the price?

let me give some hypothetical; let's say Canva decides to cut cost by cutting the amount of storage users could have (e.g. quota, or lifetime):

- if they had a sunk cost CAPEX of 100 PB of storage, all they could do is sell the storage servers for a fraction of the original cost (and folks like me snapping up pre-owned server equipment for said fraction of cost for our home labs).

- on the other, since S3 is OPEX (not sure if they are locked in to 1 year billing cycles? but still better than 3-5 year CAPEX runs), they could cut the cost of that storage "immediately".

Similarly, if they needed to expand, CAPEX means placing an order for the storage, waiting for delivery, racking, burning in etc, versus just paying almost "instantly" for more storage.

So long as they are still earning money, why wouldn't they stick to this OPEX model?

Not saying this is good or bad, it depends on your business model, ultimately.

> So long as they are still earning money, why wouldn't they stick to this OPEX model?

The suggestion was to move to a different / cheaper provider. You can stick to OPEX.

> Cloudflare R2 charges zero egress fees.

The scarier part is Canva actually does use Cloudflare CDN, so it's not so much the cost of adding a new provider. It's just that the CDN team that manages the Cloudflare account is likely a different team :)

Too much isolation, silos and politics...

The much more likely answer is that R2 is too new compared to when they started (announced in 2021) and switching from S3 to it is probably not trivial at Canva's scale.

“we store more than 230 Petabytes of data in Amazon S3, with our single largest S3 bucket coming in at a whopping 45 Petabytes”

How did they get so much data?! They say they only have 75 million stock photos. They must have tens of petabytes that are just dark.

And now it's VC-fashionable to have S3 as "object store" for "serverless" databases. "offload to S3!"

They could optimize this by only egressing they most recently used petabytes of data for some value of “recently”.

what's the durability of R2?

How would you measure that realistically? How durable is Backblaze or Azure?

Cloudflare has been reliable on storage adjacent uses (caching) for a long time. Presumably you're more likely to be the cause of errors than R2 is by a few orders of magnitude.

well you don't measure it, you take it from their sla

What I mean is that if you have a claim for eleven 9's of durability for a product that's 3 years old you have to take their word for it.

You can't really check the SLA as being realistic

eleven 9's apparently

Do you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.

I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation.

Also, it seams that it's 230PB in total, not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space.

https://wasabi.com/blog/on-premises-vs-cloud-storage/

This is a marketing pitch for a cloud storage company and it shows:

> - System hardware: $500,000

For one PB? Gold plated HDDs and cases I guess…

> Assume 15% annual system maintenance over five years

Do not run your hard drive cluster on the same floor as vibration-inducing machines, folks. /s

According to Backblaze, even a 8 year-old hard drive is still twice more reliable than this, and 15% failure rate is what to expect over 6 years of consecutive use!

> Assume the space needed to store 1PB is between 25%-50% of a standard rack (42U)

You can fit almost 10PB in a rack though, so that's another figure that they've inflated.

> Assume personnel costs to manage a 1PB system are 0.5 IT/storage admin FTE

This one is probably true (and it's even an underestimation) if you have only 1PB of data, but it's not going to scale linearly: you're never going to need 130 FTE for 260PB.

Overall, if a supplier (who's going to use the same kind of off-the-shelf tech as you would) claims that his price is 4 times cheaper that what it would cost you to run it by yourself, he's probably just lying to sell you stuff, just saying…

Can you fit 10pb in a rack, sure if you want it all flash. You also need double that for mirroring, or quadruple that for raid 10. Double that if you want a mirrored copy of that array too. If you don't have high performance needs you can get by with slower hard drives. You also have the 25gb,40gb, or 100gb top of rack switch. Well you want two of those for redundancy. It starts to quickly add up. Annual system maintenance could be lower but is still a cost. Number 1 reason I'm in a datacenter is to swap a dead drive.

Oct 2020, StorONE announced a 1PB All-Flash array for $499k.

Other examples which are close to their pricing. https://www.reddit.com/r/storage/comments/vdr6ql/pricing_exa...

> Number 1 reason I'm in a datacenter is to swap a dead drive.

No doubt about that, when you have tens of thousands of drive you end up replacing some of them all the time, but not 15% of them every year…

> Oct 2020, StorONE announced a 1PB All-Flash array for $499k.

With Optane and all, right? The kind of things that's a massive overkill if your workload allows you to use Glacier…

> Other examples which are close to their pricing.

In that list, the only offers that are close to this pricing (but still almost 20% cheaper) also includes several years of support (which you probably don't want if you're Canva's scale).

> Oct 2020, StorONE announced a 1PB All-Flash array for $499k.

mind you the price of flash storage has dropped significantly since 2020.

Hard disks are still being used, but having all flash systems is not uncommon.

140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.

Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.

The last place I worked that had that kind of capacity, I had access to the lobby of that building and that was about it. The server room I could access had space for <wiggles fingers> maybe 15-20 racks and in fact had maybe 7, which is probably why it was always cold as fuck in there and I didn’t like being in there much.

Then there was a contract which had a server room twice the size and mostly contained an AS/3<mumble> and a couple racks of Intel hardware. You could probably get a ping pong table into that one. Giant rooms that are mostly white and too cold are a little freaky, and reminiscent of 2001.

There have been a number of stories of people not being able to utilize their floor space for more servers because they ran out of space on the roof for AC units. At one point I recall Google was working on power dissipation because they had hit the max electrical code, and so no one would agree to run more power into their buildings. More servers meant more compute per watt and per ton of chillers.

AWS has a talk about things like this. They were, no idea if they still do, setup an AZ based on MWH figure for most efficiency because any higher the power company doesn't like it. Our local DC the water cooling and refrigerant cooling is larger than the 5 generators they have. I've crammed a lot of servers in my rack and I'm out of power plugs but still under 2kwh of what we are on contract for. Dense servers have really messed with datacenters.

> I don't know many places you could just casually ask for 100+ racks.

That's obviously not something you can do on a whim, but at the same time we're talking about a seven figures investment plus the time to build the in-house team to manage this, so it's not going to happen overnight anyway.

Reply on news.ycombinator.com