Sam Altman Wants $7T

Previously: [0](149 points, 5 days ago, 208 comments) [1](56 points, 4 days ago, 61 comments) + at least 6 other posts (low chatter). Tech Meme ride home podcast it last Friday

[0]: https://news.ycombinator.com/item?id=39310563

[1]: https://news.ycombinator.com/item?id=39315786

What isn't really discussed here (and I wish was!) is the corollary to the price scaling of successive generations: the outcome isn't proportionate to the increased cost. GPT-4 cost thirty times as much as GPT-3 to train, but it isn't thirty times better. It's not even twice as good! I suspect the majority of people using an OpenAI product are still using GPT-3.5 in the free tier and are mostly satisfied with it, because the basic proposition — plausibly fluent English text — is still there, and GPT-4's improvements aren't needed for most of the use cases an LLM satisfies.

Following this out, you have the prospect of this company burning multiple trillion dollars to create a foundation model that is slower than GPT-4 and possibly only 5-10% better. The only way this makes any sense financially is if you think that extra 5-10% takes us to some kind of AGI that solves all of the reasons GPT-4 hasn't put everyone out of a job. I suspect OpenAI doesn't think this, but maybe "training another model" is easier to pitch than whatever the process would be to find product-market fit for their current model.

The 7 Trillion figure seems to come from a single source reporting on Altman's discussions with the UAE on a large chip making venture to boost the world's fabrication and deployment capacity for AI compute. It is reported to cost between 5-7 trillion.

So a single source has the high number 7 trillion, which doesn't include over how long this amount will be invested, the structure of this initiative, etc. I think people read this and think the gist is "Sam Altman is requesting 7 trillion dollars personally for OpenAI to build chips right away" like it's another funding round.

I think the general premise is that he anticipates that much money will be needed to get to ASI compute and he's citing the amount now so people are apprised of the scale. Until more explicit reporting emerges the speculation seems to be more revolving around people's seeming dislike/disillusionment with AI and its impact on tech than the actual facts.

$7T raise implies like a $700T valuation right? Just a casual 7 years of world GDP.

At some point it might be cheaper just to let humans continue to do whatever work we imagine GPT-17 will automate.

People have been scared of AGI for decades, such that it would take over and destroy humanity. Hail our robot overlords! My take is that it will do that indirectly by destroying the climate first because of the energy requirements. I find it chilling to imagine the possibility that any inevitable successful AGI would require humans to first destroy themselves. As in we are setting up the precondition for it. Just like how dinosaurs were wiped out so that the homo sapiens can evolve.

In "The Population Bomb" (1968) Robert Ehrlich claimed that within 10 years there would be catastrophic famines because of exponential population growth.

A lot of people have blind faith in just extrapolating numbers out into the future with no context.

For comparison's sake: to regenerate the data that enabled AlphaFold, you'd have to spend about $20 billion.

Source: https://www.nature.com/articles/s41587-023-02103-0

>A major distinguishing factor of the sciences (specifically, biology, chemistry and physics), as compared to AI fields such as natural language processing and computer vision, is the relative lack of publicly available data suitable for these domains (see Box 1). There are simply vastly fewer data existing in the sciences, and these are often siloed by academics and companies. Acquiring appropriate scientific data for AI typically requires not only highly trained humans, but also high-end facilities with expensive equipment, making for an overall costly and slow endeavor compared to humans simply going about their day by adding to the vast trove of images, text, audio and video on the World Wide Web. As one example, it has been estimated that “The replacement cost of the entire PDB [Protein Data Bank] archive is conservatively estimated at ∼US$20 billion”1 —these are the data used by DeepMind to train AlphaFold2.

Have LLMs done anything to stimulate the world economy or improve peoples lives? As far as I can see the only thing its done so far is drain energy and money.

As history has taught us sometimes the impossible happens in an instant and sometimes the "should be easy" takes millennia to solve. The "AI" we have today is definitely usable in very narrow use cases versus the just paper-worthy "AIs" of the past however there is nothing to suggest a seven trillion endeavor will have even a marginal utility.

The biggest mistake people make when looking at a graph that goes up is assuming that it will continue to go up.

For context, US National Defense budget is $1.5T/year.

An aircraft carrier costs $13B. For another $10B you can put 100x F35s on it.

The market cap of the entire Magnificent Seven is $12T.

The impression I got from the WSJ article is that he wants a 7 trillion dollar investment in chip manufacturing world wide, not that he personally wants to raise 7 trillion dollars. If you put it that way, it's not that unreasonable at all.

That 7T to replace future people’s jobs could also go to producing the people to do those jobs. But I guess it’s too hard to make money investing in people.

Can someone in AI spice chime in on this:

Let's say we somehow have $7T to train GPT-7 or an equivalent model. Is there even enough content in the world for this? Or is the training effort completely unrelated to the amount of needed content?

I assume $7T is a commitment spread over many (10?) years, which is just mere 700B a year. Now get 100 largest companies/governments/funds invest just a 7B a year each and you will have it, ez-pz.

I would rather own 1/7th of all the houses in the US then whatever he is up to.

At one point yahoo was supposedly worth more than the entire 300 million person economy of indonesia and amazingly, it didn't last.

7t is ridiculous. maybe drug induced uophoria?

Asking for 7T is ridiculous. He needs to prove the product step-by-step, not jump three steps ahead of what exists.

So he need more horses? Maybe spend half that budget to deal with any copyright violations.

Related source story discussion from a few days ago:

Sam Altman Seeks Trillions of Dollars to Reshape Business of Chips and AI

https://news.ycombinator.com/item?id=39310563

I don't think even SoftBank would go for that.

But... if... run multiple cities off the waste heat from the compute. Easy to see the chain of logic which runs from there to the Dyson sphere made of computronium.

We live in interesting times!

Asking for $7 trillion dollars is simply how you go about getting $700billion.

$700billion looks sane next to $7T.

However if you START asking for $700B then people will say you are crazy.

smelling like weird crypto hype all over again. I don't think anyone could seriously claim gpt-4 is 30x better than gpt-4.

"Sam Alman Wants $7T (astralcodexten.com)"

Maybe I'm missing some play on words, but that should be Altman, no?

If he wants to actually do something like dethrone NVIDIA in the next 10 years, $7T sounds about right.

This is absurd. The increasing amount of compute and energy it is taking to training subsequent generations of models is all you need to look at to see that this is not and never will be the path to AGI. This is a slightly more useful boondoggle than BitCoin, but will more detrimental to society by orders of magnitude as these ersatz "good enough" AI tools put countless people out of jobs and jeopardize safety when these "AI" tools are put in the loop of critical systems. You all have been warned.

Isn't he just saying that people need to bring more than just money to the table?

The "Trough of Disillusionment" can't come fast enough for this...

$7 trillion would make OpenAI the 3rd largest economy for that year

nice writeup, pointing out the tension between quick scaling and safety.

I think a bigger problem with this is more simple: there's a pretty critical assumption that the GPT-6 and GPT-7 models will need orders of magnitude more compute.

On the one hand, that was true for GPT2 > 3 > 4. On the other hand, history is rife with fallacious predictions about technology development and resource usage, all the way back to Malthus. Typically, as the human race focuses on a problem, whether that be farm efficiency or AI or anything else, people invent new approaches that invalidate simplistic forecasting about resource usage.

In this context specifically, I think there's a lot of promising research with different types of model architectures and multimodal models, to the extent that AGI may not be a path paved through brute forcing transformer training for LLMs. In fact, I believe Sam Altman said this himself last year!!!

Also, from a pure business perspective: it's one thing to spend $7T and build a bunch of factories. It's quite another thing to beat Jensen at that game, who has a legendary history of beating the odds when his company was headed for failure by constant process innovation and strategic moves -- this podcast episode saga is a great listen: https://www.acquired.fm/episodes/nvidia-the-gpu-company-1993...

I think Sam is thinking too much like a VC here -- raise as much as you can and take your % management fees -- and I would not be eager to invest so much into brute forcing a hardware business at scale. I'd much rather invest that into NVIDIA or AMD, who have the battle scars to know how to build hardware profitably.

I thought I left a spare trillion around here somewhere...

Is this peak tech hubris?

GPT-7, a.k.a. SkyNet.

"My current impression of OpenAI’s multiple contradictory perspectives here is that they are genuinely interested in safety - but only insofar as that’s compatible with scaling up AI as fast as possible."

I'd rewrite this as "...they are genuinely interested in safety - but only insofar as it doesn't get in the way."

i do too

Sam Altman seems to want power and attention. If mankind has $7t to throw away it would be far better spent moving the entire world to renewable energy and renewable powered transportation.

Unless his ultimate goal is to leave earth to our AI powered overlords, “AI” is absolutely meaningless if the planet is uninhabitable for humans.