Hacker News Reader

Probably a better candidate for local applications? Guessing this was a trial balloon against larger models coming though.

One big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model

* as in matches the token that the larger model would output

It would be better if they also had a 70B model for this.

They would need the same vocabulary, etc. What else?

According to their website, it's "Optimal for: low latency, text summarisation, classification, text completion, code completion"

7B models are small enough to be usable on a smartphone, so a local handheld assistant sounds like a use case.

Are they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.

"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.

"usable" is not the same as practical.

Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.

Try MLC-LLM. Its not as bad as you'd think.

In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.