Hacker News Reader

Comment by DEDLINE | Hacker News Reader

I wonder if Meta would ever try to compete with AWS / MSFT / GOOG for AI workloads

FB does not have the flywheel of running data centres - all three of those mentioned run hyper scale datacentres that they can then juice by “investing” billions in AI companies who then turn around and put those billions as revenue in the investors

OpenAI takes money from MSFT and buys Azure services

Anthropic takes Amazon money and buys AWS services (as do many robotics etc)

I am fairly sure it’s not illegal but it’s definitely low quality revenue

Sounds like it's free equity at the very least

How is it free equity? Spending money to invest it somewhere involves risks. You might recover some of it if the investment is valued by others, but there is no guarantee.

You do not need cash in hands to invest. Instead, you print your own money (AWS credit) and use that to drive up the valuation, because this money costs you nothing today.

It might cost tomorrow though, when the company starts to use your services. However depending the deal structure they might not use all the credit, go belly up before credit is used or bought up by someone with real cash.

Facebook has more datacenter space and power than Amazon, Google, and Microsoft -- possibly more than Amazon and Microsoft combined...

[citation needed]

Unless you've worked at Amazon, Microsoft, Google, and Facebook, or a whole bunch of datacenter providers, I'm not sure how you could make that claim. They don't really share that information freely, even in their stock reports.

Heck I worked at Amazon and even then I couldn't tell you the total datacenter space, they don't even share it internally.

You can just map them all... I have. I also worked at AWS :)

This would be an interesting dataset to use for trading decisions (or sell to hedge funds).

But I wonder how much of their infrastructure is publicly mappable, compared to just the part of it that's exposed to the edge. (Can you map some internal instances in a VPC?)

That said, I'm sure there are a lot of side channels in the provisioning APIs, certificate logs, and other metadata that could paint a decently accurate picture of cloud sizes. It might not cover everything but it'd be good enough to track and measure a gradual expansion of capacity.

I’m not sure mapping VPCs is super helpful - the physical infra is fairly distinct.

AWS has also disclosed 20 million Nitro adapters have been deployed, so you can do some backwards napkin math from that.

Mapping as in.. drawing the outlines of buildings and computing the square footage yourself?

Yep.

Then you should be aware that, for the longest time, Google was against multiple floors, until they suddenly switched to four floors in many locations:

https://www.datacenterfrontier.com/cloud/article/11431213/sc...

A decade ago, there was a burst in construction and in some places the bottleneck was not getting the machines or electricity, but how fast they could deliver and pour cement, even working overnight.

Yep, I am aware, I have a square footage multiplier for their multi-story buildings.

But how can you know how many floors they have? And where are you getting the list of buildings from? And what makes you think your list is complete?

Also how do you know their efficiency? Google might have less space but also a way to pack twice as much compute in the same place.

Like I said, this is impossible to know without a lot of insider information from a lot of companies.

Well, they tell you for one: https://datacenterpost.com/scaling-up-google-building-four-s... We also have Google Street view, etc.

Of course, it's all estimates. You can get fancier and count generators and transformers and stuff too.

I don't think so, AWS hasn't disclosed this numbers, like datacenter spaces occupied, so how do you know.

I have mapped every AWS data center globally, and I worked at AWS.

Facebook publishes this data.

I have zero evidence, but this seems extremely unlikely. Do you have more than zero evidence?

Meta can use all their datacenter space while Amazon, Google, and Microsoft datacenter space is mostly rented.

To date, facebook has built, or is building, 47,100,000 sq ft of space, totaling nearly $24bn in investment. Based on available/disclosed power numbers and extrapolating per sqft, I get something like 4770MW.

Last I updated my spreadsheet in 2019, Google had $17bn in investments across their datacenters, totaling 13,260,000 sq ft of datacenter space. Additional buildings have been built since then, but not to the scale of an additional 30mil sq ft.

Amazon operates ~80 datacenter buildings in Northern Virginia, each ~200,000 sq ft -- about 16,000,000sq ft total in that region, the other regions are much much smaller, perhaps another 4 mil sq ft. When I'm bored I'll go update all my maps and spreadsheets.

But Google built data centers aren't the only data centers google is running their machine fleet in...

Yeah, Google buys servers in public datacenters like those from Equinix. One "region" needn't be one datacenter, and sometimes AWS and GCP will even have computers in the same facility. It's actually quite annoying that "region" is such an opaque construct and they don't have any clear way to identify what physical building is hosting the hardware you rent from them.

Those are almost lost in the noise, compared to the big datacenters. (I've been inside two Atlanta facilities, one leased and one built from scratch, and the old Savvis one in Sunnyvale).

At this point Power Companies (ala PG&E, etc) should be investing in AI companies in a big way. THen they make money off the AI companies to build out power infra - and vice versa.

I am surprised we havent heard about private electrical grid built out by such companies.

Surely they all have some owned power generation, but then if they do, the local areas where they DO build out power plants - they should have to build capacity for the local area, mayhaps in exchange for the normal tax subsidies they seek for all these large capital projects.

Cant wait until we pods/clusters in orbit. With radioisotope batteries to power them along with the panels. (I wonder how close to a node a RI battery can be? Can each node have its own RI?) (sas they can produce upto "several KW" -- but I cant find a reliable source for max wattage of an RI...)

SpaceX should build an ISS module thats an AI DC cluster.

And have all the ISS technologies build its LLM there based on all the data they create?

Does the square footage take into account multiple floors? What's the source? It can be misleading, because you don't know the compute density of what's inside. Using just public data, power is a more accurate proxy. Until at least 5-6 years ago, Google was procuring more electricity than Amazon. Before that, it had a further advantage from lower PUE, but I bet the big names are all comparable on that front by now. Anyone that has worked at several of them can infer that FB is not the largest (but it's still huge).

As for the dollars, were they just in 2019 or cumulative? The Google ones seem low compared to numbers from earnings.

Google certainly has more compute density than Amazon, the numbers I was able to find from the local power company was 250MW at Council Bluffs back in 2015 or so.

Amazon builds out 32MW shells, and the most utilized as of 5 or 6 years ago was 24MW or so, with most being much less than that.

I updated my map for AWS in Northern Virginia -- came up with 74 buildings (another source says 76, so i'll call it directionally correct). If I scale my sq ft by ~5% to account for missing buildings, we get 11,500,000sq ft in the northern virginia area for AWS.

I'll finish my other maps and share them later...

NVidia also invests in their AI customers.

What do you mean? Could you elaborate please? Enumerate some deals so I could read more about it?

Neither did AWS when they started. They were just building out data centers to run their little book website and decided to start selling the excess capacity. Meta could absolutely do the same, but in the short term, I think they find using that capacity more valuable than selling it.

> Neither did AWS when they started. They were just building out data centers to run their little book website and decided to start selling the excess capacity.

This is a myth. It simply isn't true. AWS was conceived as a greenfield business by its first CEO. Besides, S3 and SQS were the first AWS services; EC2 didn't appear till a few years later. And it wasn't built from excess Amazon server capacity; it was totally separate.

Such barter deals were also popular during the 00s Internet Bubble.

Here more on the deals (2003):

https://www.cnet.com/tech/services-and-software/aol-saga-ope...

Popular names included AOL, Cisco, Yahoo, etc.

Not sure if Amazon’s term sheets driving high valuation are nothing but AWS credits (Amazon’s own license to print money).

Meta could build their own cloud offering. But it would take years to match the current existing offerings of AWS, Azure and GCP in terms of scale and wide range of cloud solutions.

aww, those existing offerings are overcomplicated as hell, a fresh look could yield substantially simpler cloud developer experience and this would compete well against those other cloud offerings on simplicity alone

The real question is: why aren't they? They had the infrastructure needed to seed a cloud offering 10 years ago. Heck, if Oracle managed to be in 5th (6th? 7th?) place, Facebook for sure could have been a top 5 contender, at least.

because meta sucks at software, documentation and making sure end user products work in a supported way.

Offering reliable IaaS is super hard and capital intensive. Its also not profitable if you are perceived as shit.

>because meta sucks at software

Google started a cloud and their user-facing software is atrocious. Compared e.g. Angular to React, Tensorflow to Pytorch.

Why would you prefer Pytorch to Tensorflow/Keras?

Tensorflow and keras have gotten better, but pytorch historically had better flexibility than keras and was much easier to debug/develop in than tensorflow.

Because it's not their business, they're not good at it and probably the ROI is not worth it.

Also how exactly they would do it, they don't have enough infra for renting, they would need to x10 what they have now.

Because they make more money using their servers for their own products than they would renting them to other people. Meta has an operating margin of 41% AFTER they burn a ton on Reality Labs, while AWS has a 21% margin with more disciplined spending. Social media is a more profitable business than infrastructure.

Does Meta make money from anything other than ads? It's not a dismissive question. I'm curious if social media implies anything other than ads.

> Advertising (over 97.8% of revenues): the company generated over $131 billion in advertising, primarily consisting of displaying ad products on Facebook, Instagram, Messenger, and third-party.

https://fourweekmba.com/how-does-facebook-make-money/

And then there's sales. All of those three - and more you haven't considered, like the Chinese mega-IT companies - spend huge amounts on training, partnerships, consultancy, etc to get companies to use their services instead of their competitors. My current employer seems all-in on Azure, previous one was AWS.

There was one manager who worked at two large Dutch companies and sold AWS to them, as in, moving their entire IT, workloads and servers over to AWS. I wouldn't be surprised if there was a deal made there somewhere.

For consumers, AI could just be stateless "micro service". Meta already has enough surfaces where customers can interact with AI.

I think Meta have avoided doing this because it would complicate their business priorities. They don’t really do B2B.

What do you mean by “they don’t do B2B”? They sell ads to companies, don’t they?

Reply on news.ycombinator.com