Hacker News Reader

It would be cool to understand when the system will use one or the other (the ~3 billion on-device model or the bigger one on Apple servers).

View on HN

Conceivably, they don't have precise answers for that yet, and won't until after they see what real-world usage looks like.

They built out a system that's ready to scale to deliver features that may not work on available hardware, but they're also incentivized to minimize actual reliance on that cloud stuff as it incurs per-use costs that local runs don't.

Yeah this is probably right. If it works well enough during real-world usage it will be using the on-device model, if not then there is the bigger one on the servers. There is also GPT-4o, so they have 3 different models to use depending on the task.

if you have ever used a 3 billion or 7 billion parameter model you know that they are really bad at text generation, so this will be done in the cloud

Reply on news.ycombinator.com