Hacker News Reader

Comment by cubefox | Hacker News Reader

There are now at least three ways to accelerate machine learning models on consumer hardware:

  - GPU compute units (used for LLMs)
  - GPU "neural accelerators"/"tensor cores" etc (used for video game anti-aliasing and increasing resolution or frame rate)
  - NPUs (not sure what they are actually used for)

And of course models can also be run, without acceleration, on the CPU.

View on HN

An "NPU" is a matrix multiplier accelerator. It removes some general-purpose stuff that GPUs provide in favor of more "AI"-useful units, like support for values a byte or smaller (i.e., FP4, INT4, etc.).

All three of them accelerate matrix multiplications actually.

any thing that computes matmul faster than by hand technically accelerates matmul - so what's your point?

I think NPUs are often aimed at efficient matmul performance. Not all implementations are significantly faster than vector units in the CPU, but they use much lower power. Gpu acceleration is typically much faster than the CPU, but also higher power.

At least for desktop gaming, the tensor cores are in the GPU compute units (SM), same as for the big data center GPUs.

It seems ARM believe it makes sense to go a different route for mobile gaming.

From what I can tell, NPUs are mostly being used by Microsoft to encourage vendor lock-in to the MicrosoftML/ONNX platform (similar to their DirectX playbook).

They're used a lot on mobile. Apple uses their "neural engine" NPU to power their on-device ML stuff and Samsung does something similar in their Exynos processors. Apple also exposes the NPU to developers via CoreML.

Extension spaghetti is hardly any better, just because it says the same API name on the tin.

Google and Apple have been doing NPUs for a while now.

Yeah, and the NPUs have displaced approximately 0% of the hardware demand for Nvidia products. Real snipe chase they've commit to.

Extension spaghetti is fine, I'd much rather end up with AI acceleration being handled like Vulkan than suffering a fate like Metal or DirectX.

Failure to understand why developers chose CUDA, is exactly why NVidia keeps selling.

Same applies to proprietary 3D APIs.

There is a reason why only FOSS devs make such big fuss out of APIs, while professional game studios keep talk about how to take each hardware to its limits at GDC, since 8 bit heterogeneous home game systems.

Reply on news.ycombinator.com