Hacker News Reader

Comment by FL33TW00D | Hacker News Reader

I described this system word for word 2 years ago, glad to see it come to fruition on the only software stack integrated enough to do it.

https://fleetwood.dev/posts/a-case-for-client-side-machine-l...

View on HN

Thanks for posting the article.

Some predictions in the article:

1. Partial client side execution of the models, then main model execution in cloud. (Example: text encoder for Stable Diffusion runs local, UNet runs cloud)

2. Local LoRAs for “dynamic ML”

3. WebGPU will be new standard to unlock models in the web

Even though the article had some good calculations and overview, I am not sure word-for-word is how I would describe its relationship to the OP.

If you described Speculative Decoding, 3bit Quantization, and Adapter based weight selection all two years ago then your AI Lab must be so sota /s

I am curious, what do you mean by "Adapter based weight selection"?

That's apparently Apple's term for LoRA[1].

[1] https://arxiv.org/abs/2106.09685

Sort of but Adapters allow for multiple weight adjustments (think loras) for specific skills so it is more like extra optimized mixture of experts or multi agent approach. They have a slide with adapters listed like summarization, prioritization, tone (happy, business, etc), editor, etc) -- this is not to be mixed up with Intents which is how on device apps publish their capabilities to the Intelligence system for real npu os level multi agent tool use.

Reply on news.ycombinator.com