If that were the case, why not charge more?

The NPUs integrated into CPU SoCs are very small compared to even integrated GPUs, much less discrete or datacenter GPUs.

NPUs seem to be targeted towards running tiny ML models at very low power, not running large AI models.

I keep idly wondering what would be the market for a plug and play LLM runner. Some toaster sized box with the capability to run exclusively offline/local. Plug it into your network, give your primary machine the IP, and away you go.

Of course, the market segment who would be most interested, probably has the expertise and funds to setup something with better horsepower than could be offered in a one size fits all solution.

I am working on a larger project about containers and isolation stronger than current conventions but short kata etc…

But if you follow the podman instructions for cuda, the llama.cpp shows you how to use their plugin here

https://github.com/ggml-org/llama.vscode

> if coding agent achieved another step jump in abilities ~ 1trn+ marketcap

Do you want to walk us through that math?

Except that it keeps getting lawyers into trouble when they use it.

https://www.reuters.com/legal/government/judge-disqualifies-...

I don't know how it is in other countries, but in the UK using LLMs for any form of paid legal services is hugely forbidden, and would also be insanely embarrassing. Like, 'turns out nobody had any qualifications and they were sending all the work to mechanical Turks in third world countries, who they refused to pay' levels of embarrassing.

I say this as someone who once had the bright idea of sending deadline reminders, complete with full names of cases, to my smart watch. It worked great and made me much more organised until my managers had to have a little chat about data protection and confidentiality and 'sorry, what the hell were you thinking?'. I am no stranger to embarrassing attempts to jump the technological gun, or the wonders of automation in time saving.

But absolutely nobody in any professional legal context in the UK, that I can imagine, would use LLMs with any more gusto and pride than an industrial pack of diarrhoea relief pills or something - if you ever saw it in an office, you'd just hope it was for personal use and still feel a bit funny about shaking their hands.