Hacker News Reader

Aren't small local models worse efficiency-wise? It means that every person must have a powerful enough machine to power a small model, and we are very, very far away from that.

The best solution, from an efficiency point of view, is to use smaller models on datacenters, requiring much less of them.

View on HN

There's an efficiency sweet spot where hardware that people have anyway gets a higher percentage of load.

MacBooks have a lot of memory and a lot of FLOPs. They mostly sit unused all day. Yes, the excess energy use will be higher than a GPU in a datacenter doing the same work, but you have to generate an absurd amount of tokens before the dollar-efficiency catches up with the MacBook.

You need to have a 3k dollar machine available though, I think you are overestimating how many people have access to it

Reply on news.ycombinator.com