The Inference Shift – How Cheap Chips Could Put Frontier AI in Everyone's Hands
substack.com
substack.com
What paper? This is slop.
No, BitNet not requiring multiplication will not put a foundational model in your pocket. It would be nice for power if tinary models had scaled, but since it requires roughly 3x the parameters of a similarly capable model, the memory bandwidth does not scale down nearly as well.
The real trick is that a classic LLM is not useful in the scenarios the author proposes. The hypothetical livestock vet is far better served by her books and a phone call to a university ag extension to confer with colleagues than an LLM disconnected from the internet that will hallucinate nonsense.
But anyway, isn't that why the author also brings up matmuls, MLA/MoE, and post training quants, KV optimization? That's all about targeting the memory issue that is brought up. Also, memory is secondary to needing a 2nm-3nm fab to put out chips that need thousands of watts. With HBM getting faster it makes memory solvable. Makes these boxes more expensive but the hurdle of big wafers of silicon and energy requirements get side stepped.
It isn't resting on a single tech but it looks like 5 techs that do exist and if combined could threaten cloud inference companies. The article straight up says it doesn't know if this will scale, but what if it does?
As far as the vet analogy goes, I think it's it's a little too utopian and dramatic. But I have made a RAG system with my own graph set up on a large set of technical books like in the example. It has to give me a reference to the texts. So far it has not hallucinated and I check the source texts every time. Saves me a lot of time. Just believing models without giving it technical info is dumb for sure. But again that isn't what the article stated, it called out having vet books as a technical source. It's what actually made me read through it all because I've done that exact thing with my own field.
Also just because it can run offline doesn't mean it has too. No reason one of these boxes can't be connected to the internet only that it could. The possibility of it running internet free is a bonus not a requirement.