Hacker News Reader

Comment by faeriechangling | Hacker News Reader

Overhyped how? Overhyped like self-driving, in terms of the timelines? Or overhyped in terms of the eventual societal impact?

I think people seriously underestimate how much of a struggle driving down the cost of compute/$ is going to be. Every incremental gain in memory and microprocessors is getting exponentially more expensive to develop. This is also a (relatively) high interest rate environment which do not tend to favour long term speculative investments. A lot of people aren't fully aware of how OpenAI and GitHub CoPilot are bleeding red ink from the get-go.

I honestly feel like people are really sleeping on the potential still. Most people I know tried it a few times around last Christmas and got bored with it and quit but I never stopped using it.

View on HN

> struggle driving down the cost of compute/$

Actually I feel that utilization of existing hardware is very poor.

For instance, in LLM land, continuous batching is a thing that just starting to proliferate (at least in open source). Many are still doing training/inference in vanilla pipes in PyTorch eager mode, without even a simple `torch.compile` or aggressive quantization, much less a more advanced ML compiler. Architecture optimizations beyond llama's base architecture (like newer attention schemes used during training) are not really picked up yet.

I can go on and on, and go on about Stable Diffusion as well, but what I am saying is there is tons of unpicked low hanging fruit, that's unexploited because... everyone is in a rush to just publish or deploy something, I guess?

Also, Nvidia's margins are insane. The voltage/clock band they run in is insanely inefficient. I hope that is not sustainable, because sane margins/clocks with no code/hardware advancement would bring costs down a ton.

Can you explain more about the voltage/clock band please? Do you mean that they are very energy inefficient?

Yes.

Basically, power usage increases quadradicaly with clockspeed. Running a little faster takes a lot more power.

Nvidia runs GPUs smaller than an Apple M1 Ultra at 250-450W. Thats crazy for a single chip! They could sacrifice a tiny bit of performance and run with much less power... But they don't.

The Cerebras CS2, for reference, has a much lower power density and runs more efficiently. Unfortunately most other AI hardware is pretty hotly clocked at the moment.

Understood, thank you. I appreciate the explanation.

Reply on news.ycombinator.com