Sam Altman Seeks Trillions of Dollars to Reshape Business of Chips and AI

But perhaps it turns out that subnets can be trained independently or swapped with semantically equivalent but qualitatively different ones. The routing network would effectively "Standardize" and could in principle be well enough understood to "hand optimize" the routing network into hardware. Or maybe back propagation has some novel physical analogue that can be exploited in scales we can access. The real question is if Altman is capable of finding the right path in the notoriously dead end filled field of chip design. His backing of helion [1] didn't bode well in my view. But with enough R&D maybe he will flail into something useful trillions is enough for a lot of flailing.

[1] https://youtu.be/3vUPhsFoniw Edit: more derisive link

Must be really hard, being only a half-billionaire and trying to keep up with Elon's "success"...

Obviously I think my company is doing this in an unique and "correct" way, but I know of half a dozen other companies founded in the past ~18 months that are focused on the memory capacity and bandwidth bottlenecks that exist... the massive failures of the previous decade do not mean that they are going to be repeated.

E.g. FB saying they want to buy 350k H100. That's just a whopping $14B price tag. With a >85% profit margin. While a fab is $20B.

Trillion? Sounds like anchoring to me. Nvidia has a market cap of $1.7T. You could literally buy NVIDIA for that. I read that as "a billion won't cut it, we need quite a few billions".

But it's not unreasonable that those hyperscalers throw in a few billion each.

Usually it's horrible business not to be best (see Intel/AMD). Because the margins are at the top. In this case though they want a whole range of products to go down in margin. Even a slightly worse chip might be worth it if it comes at a significant cost reduction. Especially if the optimal design is known!

In a sense the whole thing can fail at reaching the top or making lots of money and still succeed in bringing total cost down, potentially by 50% or more.

Is this true for FB? They just spent billions on 500,000 H100s (disclosed in their earnings report a week or two ago).

But even more importantly, it seems like the ultimate goal could be reasonably expected to drive large scale unemployment, with no clear replacement for such. Necessity is the mother of invention, but in this sort of scenario you're expecting kings of industry and politicians to put aside their greed and self interest for the sake of social good. The odds there are going to be overwhelmingly in favor of dystopia, which we substantially collectively sacrificed to achieve?

[1] - https://en.wikipedia.org/wiki/Sigmoid_function

I somehow doubt you can convince even half the liberal voterbase that that would be a good idea. let alone any of the conservative base. But I guess the US did manage to do that with the space race, so maybe the key is another cold war.

https://techcrunch.com/2019/05/18/sam-altmans-leap-of-faith/

This is just to show that GDP is not necessarily reflective of investment power.

EDIT: Typo

It's clearly a plot to put more control of world finances into the hands of the US. Nations should not expose themselves to such risks.

though even trillions is an exaggeration here. If they all collectively had 10m employees making 100k on average for 10 years, we're an order of magnitude off.

Of course someone will make breakthroughs. Everyone stands on the shoulders of giants.

As evidence - if I learn a new game, like chess, my rate of mastery per game is going to be orders of magnitude higher than an AI (which will take millions or billions of training sessions). Granted my brain has more synapses than GPT4, but it's unclear how many of those I use playing chess (I wouldn't be surprised if it's less). And regardless I don't think that anybody's arguing that adding more layers accelerates training but rather increases the peak.

So perhaps a place to start is studying cases where AIs learn much slower than humans and trying to understand better algorithms of training/reinforcement that may come closer to what the human brain does.

I mean this should be obvious but the "ability to raise trillions" is because those people think they will get X returns on the money. Your "ability to raise trillions" will quickly evaporate if you offer to throw the money into a pit and set it on fire.

If national governments offered a $10T bounty for solving global warming, then sure, you could raise $1T to moonshot it... but there needs to be an incentive.

We already know it is possible. You just need to hire the right people.

Elon launched reusable rockets into space using the same philosophy. Jobs built a personal computer likewise.

The scale and scope of what's being described seems outlandish, I think $150B would probably be enough to at least get started. that should get you two plants in a country more amenable to building and running a chip facility than the US, plus a hardware design team (to make AltmanTPU v1), a software design team (to make ATPU work with pytorch), and the business side of things.

I don't see these things being sold commercial, it's more like airplanes- a few big players put up the capital to fund the construction of large batches and then get dibs on the first good volume runs.

I think the real question here is what will ASML's involvement be, what process and design will the TPU have, what will the computational infrastructure surrounding the TPU look like, and how we are going to plumb all the necessary power to the TPU farms.

It is known very long time, that specialized AI chips are more efficient than GPUs, even when we talk about Tensor cores, because GPU is not only Tensor cores, they MUST include ramdac, all other video stuff, and network for fast communications of all this stuff, which is not of need for AI chip.

Before Tensor cores, benchmarks was about 4 times against GPUs, now better but still significant lag.

They had an edge, they lost it, this is now desperate scrambling for cash