Tangent question: what about NPU performance? I always read about CPU vs GPU, but nobody talks about NPUs; I don't even know if llama supports them.
Why do phones and laptops get equipped with NPUs if there's no use for them?
Why do phones and laptops get equipped with NPUs if there's no use for them?
OpenCL and Vulkan are supposed to gain support eventually but given their extremely constrained instruction sets and the wide variance in capabilities between different hardware models I don't imagine they're the sort of thing you'd want to target manually.
I did see one prototype someone built using private ios APIs a few months back. They were able to run a reasonable SLM about an order of magnitude slower than GPU but with a tiny power draw.