Hacker News Reader

Comment by mdp2021 | Hacker News Reader

mdp2021

> Mythic M1 stores up to 80 million neural network weight parameters directly on-chip

Which means connecting ~350 chiplets to run a Qwen 3.8 27b and over 30000 chiplets to run Qwen3.8-2.4T-A95B. Cost? Space? Feasibility?

Edit: wrong values, lost a zero...

Edit: seemingly, the M1 is only part of the whole need. With the M1, you would run a feedforward pass of the NN but use the rest of the Von Neumann architecture to manage the data. The pass in the M1 will be lightning fast, the rest still a bottleneck. The M1 is almost explicitly not for LLMs.

View on HN

The vanguard chiplets even steps down to 30M… but they do claim to have a “Mead” design [1] designed to run GPT-3 in a single chip.

3D NAND flash can indeed routinely store hundreds of GB per die, so that’s proven. The question is about all the peripheral circuitry needed. Each attention block would need its own KV cache (i.e. SRAM or DRAM somewhere), plus DAC/ADC inputs and outputs, unless they figure out a way to keep it analog all the way (really cool but unlikely).

I think this field is very interesting, at least from a technology point of view. Whether it works out or not will sadly be a matter of economics more than physics I fear.

[1] https://www.mythic.ai/mead

> unless they figure out a way to keep it analog all the way (really cool but unlikely)

Very unlikely for the connection to the cache RAM, seemingly impossible if we want to get an articulate output :)

> they do claim to have a “Mead” design [1] designed to run GPT-3 [ - i.e. to hold a 175b NN - ] in a single chip

Careful: that is the /intention/, but the chip is just the storage (and CiM) for the NN and other parts are missing - explained in the Vanguard, not explained in the Mead.

> Whether it works out or not will sadly be a matter of economics more than physics I fear

Yes, but:

-- what is not enabled today may often be tomorrow through advances, esp. in the economy of production;

-- a very great point about this product seems to be that the chips can be rewritten, the NN is not etched and static (cpr. Taalas): that makes the practical implications extremely relevant, the demand would be "screaming mob" like;

-- we have to go in that direction (of NNs in CiM) anyway, so it's just a matter of time, effort after effort we will get there.

This might end up being a strong use case for FeFET FRAM, nonvolatile, high density, and sram fast. In an alternative universe where fram infra was as widespread as flash, memory is fast, cheap, and nonvolatile. It’s just that flash built the infrastructure and climbing that ladder is nearly impossible.

I think we are at the apex of Von Neumann machines. Once there is enough money to make alternatives, they will thrive. And AI is that catalyst.

Reply on news.ycombinator.com