Building Meta's GenAI infrastructure
engineering.fb.com
engineering.fb.com
[1]: github.com/google/gemma.cpp
Plain uint8 wouldn’t allow for the same accuracy range as float8 and it’s the accuracy not the precision (which uint would win for the largest values it can represent) that counts most.
The issue with LUTs is don't you have to update the LUT itself? You can select which memory address to load up, but the LUT itself has to be differentiable maybe? TBH I'm not an expert on LUTs.
On fixed point - similarly ye you have to fix the precision ranges as well, so again I'm unsure on how one changes the fixed point numbers over time. I'll have to read more on fixed point.
Maybe 1.58bit using (-1, 0, 1) which gets rid of multiplications and just additions might be more useful, although you'll only get a 2x FLOP boost since you still need fp8 or fp16 addition.
There is also VPERMI2B [0] which operates on a 128 byte LUT.
But rest assured there's an improvement, it's not like people would be doing it if there wasn't any benefit!
“bfloat16 data type and arithmetic instructions (AI and others)”
https://eclecticlight.co/2024/01/15/why-the-m2-is-more-advan...
E5M2 is like an IEEE 754. But to compensate the smaller exponent, "E4M3’s dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs".
Some people reported E4M3 is better for the forward pass (small range, more precision) and E5M2 is better for the backward pass (bigger range, less precision). And most implementations have some sort of scaling or other math tricks to shrink the error.
[0] FP8 Formats for Deep Learning (Nvidia/ARM/Intel) https://arxiv.org/abs/2209.05433
Training AI models costs a fortune, but so far it's been just front-loading costs in hopes of a windfall. We'll see what actually happens.
Its easier to spin up a business for sure -- also easier to unwind it - there not as sticky as they used to be.
The only question is - what tactic? I don't really know, but one trick I am aware of is "specifying to the vendor." In other words, the introduction of regulatory requirements that are at every step in the process a description of the most favored vendor's product. As the favored players add more features, potentially safety features, those features are required in new regulations, using very specific descriptions that more or less mandate that you reproduce the existing technology, to use a software engineer's term, bug-for-bug. If your product is better in some ways but worse in others, you might have a chance in the market - but to no avail, if the regulations demand exactly the advantages of the established suppliers.
The dream may be barriers to entry that allow high margins (“rents” if you prefer the prejudicial), but all too often these huge capital costs bankrupt the company and lose money for investors (see: WeWork, Magic Leap). It is high risk, high return. Which seems fair.
I would wager that the concept needs a bit of a refresh as historically it has referred to high capital costs for the production of a hard good though in this case there is more than just a good produced theres a fair bit of influence and power associated with the good and a ton of downstream businesses that are reliant upon it if it goes according to plan.
Uber just now turned its first profit since 2009, and I would wager that if not for the newly found appreciation of efficiency and austerity, it would still be burning through money like a drunken socialist sailor.
Classic approach required basic math. "Here is my investment, here is what I am going to charge for rent". You actually can figure out when your investment starts paying off.
This new "model" requires tall, loud, truth-massaging founders to "charm" VCs into giving away billions, with the promise of trillions, I guess. The founders do talk about conquering the world, like, a lot.
I do not know what the WeWork investors were thinking when they expected standard real estate to "10x" their money while the tenants were drinking free beer on tap. The whole thing screamed "scam" even to a lay-person.
Which gives me hope that - like the web - hardware will catch up and stuff will become more and more accessible with time
To make your own competing LLM today you need hundreds of millions of dollars, the "very expensive" of this is on a whole different level. You could afford the things you talked about on a software engineering salary, it would be a lot of money for that engineer but at least he could do it, no way anyone but a billionaire could fund a new competing LLM today.
That’s why Sam Altman makes so much noise about “safety” - OpenAI would really like a government-backed monopoly position so they can charge higher rents and capture more of that value for themselves. Fortunately, I think that llama has already left the barn.
It's true it'll still be relatively expensive - but I would propose its relatively inexpensive if people want to make it faster, and have the drive to do it :) On the other hand, capital expenditures requires large amounts of money, which also works.
I guess some general CUDA, some maths, knowing how to code transformers from scratch, some Operating systems and hardware knowledge, and the constant drive to read new research papers + wanting to make things better.
I just think as humans, if you have drive, we can do it no matter the contraints!
I don't know a lot about ML. Does anyone know if it is possible to keep training the system while it is running?
That would help a lot if you don't have the possibility to use huge training sets as a starting point.
I assure you that before Apache and Linux took over that "dot" in the .com was not cheap!
Fortunately it only really lasted maybe 1993-1997 (I think Oracle announced Linux support in 1997, and that allowed a bunch of companies to start moving off Solaris).
But it wasn't until after the 2001 crash that people started doing sharded MySQL and then NoSQL to scale databases (when you needed it back then!).
It's early. You can do LORA training now on home systems, and for $500 you can rent enough compute to do even more meaningful fine-tuning. Lets see where we are in 5 and 10 years time.
(Provided the doomers don't get LLMs banned of course!)
They mention custom building as much as they can. If FB magically has the option to 10x the compute power, would they need to re-engineer the whole stack? What about 100x? Is each of these re-writes just a re-write, or is it a whole order of magnitude more complex?
My technical understanding of what's under the hood of these clusters is pretty surface level- super curious if anyone with relevant experience has thoughts?
Its also a group effort to provide simple to use primitives that "normal" ML people can use, even if they've never used hyper scale clusters before.
So you need a good scheduler, that understand dependencies (no, the k8s scheduler(s) are shit for this, plus it wont scale past 1k nodes without eating all of your network bandwidth), then you need a dataloader that can provide the dataset access, then you need the IPC that allows sharing/joining of GPUs together.
all of that needs to be wrapped up into a python interface that fairly simple to use.
Oh and it needs to be secure, pass an FCC audit (ie you need to prove that no user data is being used) have a high utilisation efficiency and uptime.
the model stuff is the cherry on the top
For other settings, moving to something like opencue might be better (caveats apply)
To get the job he applied for a spot I'm Software Engineer applied in Machine Learning, he went through the multiple step interview process, and then when he got the job he did a few weeks of training and interviewing teams. One of the teams in charge of optimizing ML code in Meta picked him up and now he works there.
Because of Meta's scale, optimizing code that saves a few ms or watts is a huge impact in the bottom line.
In sum:
- Get a formal education in the area - Get work experience somewhere - Apply for a big tech job in Software Engineer applied with ML - Hope they hire you and have a spot in one of the teams in charge of optimizing stuff
I have a PhD in CS, and lots of experience in optimization and some in throughput/speedups (in an amdahl sense) for planning problems. My biggest challenge is really getting something meaty with high constraints or large compute requirements. By the time I get a pipeline set up it's good enough and we move on. So it's tough to build up that skillset to get in the door where the big problems are.
Feel free to DM me to learn more.
Some folks start with more familiarity in ML research and dip down as far as they need.
Other folks come from a traditional distributed systems/compilers/HPC background, and apply those skills to ML systems.
Which is a fourth of what they spent in VR/AR in a year. And Gen AI is something they could easily get more revenue as it has now become proven technology, and Meta could possibly leapfrog others because of the data moat.
Meta certainly has an edge in engineer count, undoubtedly. But I'd say they really, really want the metaverse to succeed more to have their on walled garden (i.e. equivalent power of Apple and Google stores, etc.). There's a reason they gave a hard pass to a Google partnership.
Source would be appreciated, because this is opposite of obvious. Regulations against using public first party would be a big news and I haven't heard of anything like that. They use my data for recommending feed so why not for answering my question?
Decreased ability to 'mix' it: https://www.linkedin.com/pulse/ecpms-why-first-party-data-bi...
First party data alone can't tell you whether an ad resulted in a sale, unless you own the entire process on your platform. Contrast this with what Apple has via its app store; the fees do more than generate money.
AIUI, Google required Meta to basically cede control of a partnered OS to them:
"After years of not focusing on VR or doing anything to support our work in the space, Google has been pitching AndroidXR to partners and suggesting, incredibly, that WE are the ones threatening to fragment the ecosystem when they are the ones who plan to do exactly that.
"We would love to partner with them. They could bring their apps to Quest today! They could bring the Play store (with its current economics for 2d apps) and add value to all their developers immediately, which is exactly the kind of open app ecosystem we want to see. We would be thrilled to have them. It would be a win for their developers and all consumers and we’ll keep pushing for it.
"Instead, they want us to agree to restrictive terms that require us to give up our freedom to innovate and build better experiences for people and developers—we’ve seen this play out before and we think we can do better this time around."
-- From Mark Bosworth
I don't think anyone else has gotten that group chat with AI thing so nailed.
It's not impossible. The prediction from many(not that I believe it) is that over long run modelling tricks would become common knowledge and only thing that matters is compute and data, both of which Meta has.
Also there could be a trend of LLMs for ads or feed recommendation in the future as they has large completely unstructured dataset per user across multiple sites.
IMO standalone AI companies like OpenAI might be successful by providing infrastructure to other companies, but I can’t imagine ChatGPT remaining #1 many years from now.
The web is still trending towards being a walled garden. Maybe not right now, but long term I think people will use whatever AI is most convenient which probably will be AI built into a giant company with established user base (FB, GOOG, MSFT, and Apple if they ever get around to launching - would love Siri 2.0 if it meant not needing to open the ChatGPT iOS app)
It is a real loophole in the economy. If you're a trillion dollar company the market will insist you set such sums on fire just to be in the race for $current-hype. If they do it drives their market cap higher still and if they don't they risk being considered un-innovative and therefore doomed to irrelevancy and the market cap will spiral downwards.
Sort of reminds me of The Producers.
Funnily enough Arpanet and all that Xerox stuff were like <$50 million (inflation adjusted!) total. Some real forward thinkers were able to work the system by breaking off a tiny pittance of a much larger budget.
Where as I think this more appropriately can be considered the meta PR budget. They simply can't not spend it, would look bad for Wall Street. Have to keep up with the herd.
> Funnily enough Arpanet and all that Xerox stuff were like <$50 million (inflation adjusted!) total.
How about the cost to build a fab for the Intel 8088 versus a fab that produces 5nm chips running @ 5GHz. Again, at least one or two orders of magnitude.
You don’t think earning increasing amounts of tens of billions of dollars in net income per year at some of the highest profit margins in the world at that size for 10+ years has anything to do with market cap?
https://www.macrotrends.net/stocks/charts/AAPL/apple/researc...
Roughly 30B USD per year. And what are we getting? Slightly slimmer phones and 3500USD AR/VR headsets?
I'm confused. How does your stock price, which determines market cat, affect your cashflow to fund R&D? It does not.
Thanks! (Your number is consistent with what I hear of, but I never managed to get solid sources to back them up)
I’m guessing that Meta got a sweetheart deal to help take a lot of inventory for NVidia and make commitments for future purchases.
I wonder if they will use it in RSC.
I’d point the interested at the DLRM paper [1]: that was just after I left and I’m sad I missed it. FB got into disagg racks and SDN and stuff fairly early, and we already had half-U dual-socket SKUs with the SSD and (increasingly) even DRAM elsewhere in the rack in 2018, but we were doing huge NNs for recommenders and rankers even for then. I don’t know if this is considered proprietary so I’ll play it safe and just say that a click-prediction model on IG Stories in 2018 was on the order of a modest but real LLM today (at FP32!).
The crazy part is they were HOGWILD trained on Intel AVX-2, which is just wild to think about. When I was screwing around with CUDA kernels we were time sharing NVIDIA dev boxes, typically 2-4 people doing CUDA were splitting up a single card as late as maybe 2016. I was managing what was called “IGML Infra” when I left and was on a first-name basis with the next-gen hardware people and any NVIDIA deal was still so closely guarded I didn’t hear more than rumors about GPUs for training let alone inference.
350k Hopper this year, Jesus. Say what you want about Meta but don’t say they can’t pour concrete and design SKUs on a dime: best damned infrastructure folks in the game pound-for-pound to this day.
The talk by Thomas “tnb” Bredillet in particular I’d recommend: one of the finest hackers, mathematicians, and humans I’ve ever had the pleasure to know.
[1] https://arxiv.org/pdf/1906.00091.pdf
[2] https://arxiv.org/pdf/2108.09373.pdf
[3] https://engineering.fb.com/2022/10/18/open-source/ocp-summit...
OpenAI takes money from MSFT and buys Azure services
Anthropic takes Amazon money and buys AWS services (as do many robotics etc)
I am fairly sure it’s not illegal but it’s definitely low quality revenue
It might cost tomorrow though, when the company starts to use your services. However depending the deal structure they might not use all the credit, go belly up before credit is used or bought up by someone with real cash.
Heck I worked at Amazon and even then I couldn't tell you the total datacenter space, they don't even share it internally.
But I wonder how much of their infrastructure is publicly mappable, compared to just the part of it that's exposed to the edge. (Can you map some internal instances in a VPC?)
That said, I'm sure there are a lot of side channels in the provisioning APIs, certificate logs, and other metadata that could paint a decently accurate picture of cloud sizes. It might not cover everything but it'd be good enough to track and measure a gradual expansion of capacity.
AWS has also disclosed 20 million Nitro adapters have been deployed, so you can do some backwards napkin math from that.
https://www.datacenterfrontier.com/cloud/article/11431213/sc...
A decade ago, there was a burst in construction and in some places the bottleneck was not getting the machines or electricity, but how fast they could deliver and pour cement, even working overnight.
Also how do you know their efficiency? Google might have less space but also a way to pack twice as much compute in the same place.
Like I said, this is impossible to know without a lot of insider information from a lot of companies.
Of course, it's all estimates. You can get fancier and count generators and transformers and stuff too.
Facebook publishes this data.
Last I updated my spreadsheet in 2019, Google had $17bn in investments across their datacenters, totaling 13,260,000 sq ft of datacenter space. Additional buildings have been built since then, but not to the scale of an additional 30mil sq ft.
Amazon operates ~80 datacenter buildings in Northern Virginia, each ~200,000 sq ft -- about 16,000,000sq ft total in that region, the other regions are much much smaller, perhaps another 4 mil sq ft. When I'm bored I'll go update all my maps and spreadsheets.
I am surprised we havent heard about private electrical grid built out by such companies.
Surely they all have some owned power generation, but then if they do, the local areas where they DO build out power plants - they should have to build capacity for the local area, mayhaps in exchange for the normal tax subsidies they seek for all these large capital projects.
Cant wait until we pods/clusters in orbit. With radioisotope batteries to power them along with the panels. (I wonder how close to a node a RI battery can be? Can each node have its own RI?) (sas they can produce upto "several KW" -- but I cant find a reliable source for max wattage of an RI...)
SpaceX should build an ISS module thats an AI DC cluster.
And have all the ISS technologies build its LLM there based on all the data they create?
As for the dollars, were they just in 2019 or cumulative? The Google ones seem low compared to numbers from earnings.
Amazon builds out 32MW shells, and the most utilized as of 5 or 6 years ago was 24MW or so, with most being much less than that.
I'll finish my other maps and share them later...
This is a myth. It simply isn't true. AWS was conceived as a greenfield business by its first CEO. Besides, S3 and SQS were the first AWS services; EC2 didn't appear till a few years later. And it wasn't built from excess Amazon server capacity; it was totally separate.
Here more on the deals (2003):
https://www.cnet.com/tech/services-and-software/aol-saga-ope...
Popular names included AOL, Cisco, Yahoo, etc.
Not sure if Amazon’s term sheets driving high valuation are nothing but AWS credits (Amazon’s own license to print money).
Offering reliable IaaS is super hard and capital intensive. Its also not profitable if you are perceived as shit.
Google started a cloud and their user-facing software is atrocious. Compared e.g. Angular to React, Tensorflow to Pytorch.
Also how exactly they would do it, they don't have enough infra for renting, they would need to x10 what they have now.
There was one manager who worked at two large Dutch companies and sold AWS to them, as in, moving their entire IT, workloads and servers over to AWS. I wouldn't be surprised if there was a deal made there somewhere.
Granted, HW is much harder than SW, but I would not discount Meta's ability to displace NVIDIA entirely.
Meta is a services company, their hardware is secondary and for their own usage.
Good hardware, good software support, and market is starving for performant competitors to the H100s (and soon B100s). Would sell like hotcakes.
Beyond that they might not be as stable or resilient outside of the closely curated confines of their own data-centers. In that case selling them would be more of an embarrassment.
Once you go out of your heavily curated hardware stack, the headaches multiply exponentially.
Google used to sell a search appliance-in-a-box and eventually lost interest because hardware is so high-touch.
We had a GSA for intranet search and other than the paint this was a standard Dell server. I remember not being impressed by what the GSA could do.
We also had Google Urchin for web analytics, it wasn't a hardware appliance but the product wasn't very impressive either. They then killed that and tried to get you onto Google Analytics.
They just didn't commit to these on premise enterprise products.
We had one at my company and it was widely loved- far better intranet search and domain-specific search for biotech.
It was the very definition of a hype cycle as far as I can see. Hype cycle doesn’t mean “useless and will go away”, you have the second upward curve and then productivity.
Colloquial, dismissive use of “hype cycle” does not usually mean “this will change the world but foolish things, soon forgotten, will also be done in the short term”. Though I agree a deeper understanding of the term can suggest that.
Meta has since gotten better at it- likely with lots of AI-assistance and their revenue numbers reflect this. The targeting is now likely probabilistic in that the advertiser now makes educated guesses on the best ads to serve based on limited or non-existent identity information.
So the AI efforts would have paid back by way of higher revenues.
Those GPUs are going to subsume the entire music, film, and gaming industries. And that's just to start.
I work in film. I've shot dozens of them the old fashioned way. I've always hated how labor, time, and cost intensive they are to make.
Despite instructions from the luminaries to "just pick up a camera", the entire process is stone age. The field is extremely inequitable, full of nepotism and "who you know". Almost every starry-eyed film student winds up doing drudge work for the rest of their lives. Most will never make a feature to match their ambition.
If the whole task was to simply convey my thoughts and dreams to others, why am I scrambling around to sign location rights, capture photons on expensive glass, and then smear and splice things together for months on end? This is ceremonial and soon to be anachronistic. I'm glad that whole mess is going to be replaced. It's a farce.
To phrase it another way - would you like to be hand-writing assembly on punch cards? To only gain entrance into the field with your mathematics PhD?
To speak of the liberty and the economics, why should I have to sell the rights to my idea to a studio so I can get it off the ground? Why should I have to obey the studio's rules and mind their interference?
This whole Gen AI thing is going to be the biggest liberating moment for filmmaking creatives. I know, because I am one.
And if you think any Jack or Jill can just come in and text prompt a whole movie, you're crazy. It's still hard work and a metric ton of good taste.
Art will never die. It's the human soul. It'll take more than some tech bros with GPUs to kill it.
AI is just another tool for the artist. A "bicycle for the mind" to quote Jobs, and a rocket ship for the imagination to convey my own direct experience.
Yeah, I cant wait for ChuChuTV to get the best film Oscar /s.
If you want anything good, yes. If you just want something… I reckon it'd take a week to assemble an incomprehensible-nonsense-film pipeline, after which it's just a matter of feeding the computer electricity.
Short-term, this is going to funnel resources away from the people with good taste. Long-term, it might help collapse the entire "creative industry", after which we might get some of that artist liberation stuff you're talking about – but we might just end up with new gatekeeping strategies from the wealthy and connected, and business as usual.
You don't even need AI for that.
https://en.wikipedia.org/wiki/YouTube_poop
https://en.wikipedia.org/wiki/Skibidi_Toilet
The idea that AI isn't going to be used as a creative tool too and that it won't lead to more and better art is a defeatist, Luddite attitude.
Similarly shaped people thought that digital cameras would ruin cinema and photography.
> Short-term, this is going to funnel resources away from the people with good taste.
On the contrary - every budding film student will soon [1] be able to execute on their entire visions straight out of the gates. No decades of clawing their way to a very limited, almost impossible to reach peak.
> it might help collapse the entire "creative industry"
The studio system. Not the industry.
> new gatekeeping strategies from the wealthy and connected, and business as usual.
Creatives have more ways of building brands and followings for themselves than ever before. It's one of the largest growing sectors of the economy, and lots of people are earning livings off of it.
You'll be able to follow that steampunk vampire creator that's been missing from the world until now. Every long tail interest will be catered to. Even the most obscure and wild tastes, ideas, and designs. Stuff that would never get studio funding.
As a creative, I'm overjoyed by this. My friends and I are getting to create things we never could make before [2].
[1] This and next year.
[2] Just an inspiration / aesthetic sample, but we're making a full film: https://imgur.com/a/JNVnJIn
Obviously, but you seem to be arguing that AI is just another evolution of productivity tools. You still need to have a photographer's eye while using this technology.
If you couldn't make a good composition on film, a digicam will not save you, and it definitely did not replace photographers. Perhaps lowered the barrier of entry for prosumers.
https://www.nytimes.com/2023/12/26/opinion/ai-future-photogr...
Your optimism reminds me of the optimism I had around the early internet. Power to the people, long tail, rise of the creative class, the fall of gatekeeping corporations, etc.
It was like that for a couple of years in the late 90s before power and control got vastly more centralized than before. Maybe this time it’ll be different.
With generative AI, models will be controlled by a handful of giant corporations who have the enormous corpuses (of dubious provenance) and compute ability to train them.
So it will be like last time, but even worse.
And there are other angles to consider. Apple, for one, is expressly interested in not becoming a thin client to cloud AI. They're baking a lot of inference power into their chips. If the creative class don't need their devices, that doesn't bode well for them...
FakeYou, CivitAi, WeightsGg, Comflowy, ... -- there are tons of vibrant communities to teach you everything you need to know. The tools are open source, free to use, and accessible.
This isn't hard at all once you dive in.
If you think the Luddites were defeatist, you don't know much about the Luddites.
> On the contrary - every budding film student will soon [1] be able to execute on their entire visions straight out of the gates. […] Creatives have more ways of building brands and followings for themselves than ever before.
Yet, we have no shortage of starving artists. Will AI provide them food and shelter?
This is unequivocally a win for creative expression for hobbyists, but it stands to harm professionals – at least in the short term, perhaps longer-term. It's not happening in a vacuum: the greedy are revoking livelihoods because they think AI can do it faster and cheaper (laundering appropriated hobbyist and increasingly-cheap professional labour).
> The studio system. Not the industry.
Huh, the word 'industry' has a specialised meaning in economics. Didn't know that.
Maybe, but it's never been cheaper to make a movie.
I know someone with no connections and (almost) no money which in 4 years made multiple no. 1 box-office films (obviously not in US, in a smaller country) and then got picked up by Netflix.
Said the bank teller, record producer, etc.. Plenty of cases where we've been told technology and automation would democratise the field and remove the middleman, and actually it's the opposite.
Yes, it would be nice if AI made it easy for anyone who wanted to make a great movie. That doesn't mean it's going to happen.
Something like counting an autoplaying video that ran for 3 seconds as a 'view' IIRC
[1] https://twitter.com/adamconover/status/1183209875859333120
Feels like in hind sight, maybe they were just to early to it.
I see what you did there, Meta.
https://www.reuters.com/technology/inside-metas-scramble-cat...
Interesting dig on IB. RoCE is the right solution since it is open standards and more importantly, available without a 52+ week lead time.
Sharing on Hacker News ... they now their audience.
but I suspect its not that, because Twine is optimised for services rather than batch processing, and doesn't really have the concept of priorities.
Such a large number, makes sense?
Meta's commitment to Open Source is well under calculation.
OCP is a way to rally lower-tier vendors to form a semi-alliance to keep up with super-gorilla like AWS & Google.
LLaMA has already gained much more than its cost (look at the stock price, and the open source ecosystem built surrounding LLaMA, and Google's open source Gemma models which is a proof of Meta's success).
IMHO, Meta's Open Source strategy already covered at least 5 years in prospect. That's enough to finesse a 180 degree turn around if necessary (i.e., from open source to close source)
https://www.amazon.com/Tesla-NVIDIA-Learning-Compute-Graphic...
But I do wonder how they foresee monetising this.
Whether or not you think the "value" of AI products is proportional to their performance gap vs the next closest thing or not is up to you. Very interesting PG essay I read recently talks about the opposite of this (Superlinear returns) where if you're half as good as the next competitor, you don't get half the customers, you get 0.
The “0% of the cost” part is unique to software businesses because you can copy software so cheaply
If they make AI models free to use it makes OpenAI nearly valueless, which means that they can't survive and then sell Meta's competitors a better GenAI product than Meta can make themselves.
So basically since they don't make money directly on GenAI, it makes sense for them to release it for free so no one else can have something better, so they don't have to compete on GenAI abilities with their competitors.
Since people don't want to talk to algorithms, this would result in them shunning all social media, which is a huge danger to companies in the space.
In contrast, Microsoft is spending over $10b per quarter capex on cloud.
That makes Zuck look conservative after his big loss on metaverse.
https://www.datacenterdynamics.com/en/news/q3-2023-cloud-res...
I guess people buy their vr headsets, if that counts. I'm not too familiar with what the "metaverse" entails though...
> In contrast, Microsoft is spending over $10b per quarter capex on cloud.
to service other people's work load. Its a different business.