Hacker News Reader

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

The 4090 has half the memory bandwidth, so it could not get a 5X gain, it would actually run slower on a memory bound LLM like this.

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

Yeah, it is an architectural simulation study, this is what is usually done right at the beginning before resources are allocated to go deep on idea. So in that sense it is imaginary; but this is how new ideas get incubated.

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

Yeah, a preliminary architectural study to sanity check if an idea could potentially pay off.

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

Maybe they could do something like AMD's GPU memory stacking, that is good for scaling, and of course they are using many chips not one chip..

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

Seems like HN comments have determined that the cost number is not fudged..

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit.

Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit is likely to be lower. But even a 3X is huge in this space.

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

Yep, it's a research paper in comp arch, the initial proof-of-concept study before you go and spend real money on it.

kraken12on Chiplet ASIC supercomputers for LLMs like GPT-4

They are using many chips and taking advantage of the way data flows in LLMs to make it work; so it would be cost-effective unlike Cerebras