Comment by peri-cl | Hacker News Reader

Comment by peri-cl | Hacker News Reader

Parent Full thread

Well, it depends on the task, doesn't it? "running shoes I looked at last week" / "Here's what I found in your browsing history:" doesn't need a 119 billion parameter frontier model; it's a RAG problem for the 0.6 B embedding models. That's an example Mozilla offers. Presumably to explain to their users why it's essential they hand over their last week's browsing history for this convenience (but it isn't! Hardly for that!)

I feel it's wrong to tell users that it's important and normal to relinquish all control of their—extremely personal—life history, in bulk, in plaintext, to strangers.

I agree wholeheartedly that remote server inference is super useful, and that local inference falls far short on many tasks. (I have no objection at all to Mozilla providing a cloud inference feature).

What I don't buy is that we must ask users to redraw their personal boundaries so that their most intimate life details, and remote frontier-model inference, overlap. They do not need to overlap.

You can accomplish a lot with private local inference with the smallest of models; and you can accomplish a lot on remote servers which aren't privy to everything. If some convenience is lost by not combining the two, well, so be it. I'm sure most people would agree, if all of this was laid out plainly.

You’re taking their shoe example way too literally. It’s just showing they’re catering to the average user, not an engineer. And tons of average Joe users now have very high expectations of how intelligent a LLM is because they’ve interacted with ChatGPT and the like. If there’s a super tiny model that is too dumb to do anything beyond shoe searching and they have to switch to Gemini and google AI search for anything more complex, this whole launch would be an immediate failure.

I'm of the opinion that local inference should be done to the greatest extent that is realistically possible, at the earlier time that the hardware/average user platform is capable of doing so. I personally spend a fair bit on kWh extra in my home electrical bill monthly for having a good sized chunk of local inference ability in my house, but that's not a common thing yet.

If mozilla is doing things to send users down the path of doing this externally, they need to be much more upfront and transparent with the users about where their data is going, and not bury it in some terms/conditions that only nerds will hunt for.

> "running shoes I looked at last week" / "Here's what I found in your browsing history:"

Does that need an LLM at all?

Not really, but it's probably easier to make it on top of LLM than to make specially-purposed tool for it, if we talking in terms of time-to-market effort.

SQLite FTS could have done this a decade ago. We've had good local search capabilities for two decades and they either been underused or abandoned (e.g. Google Desktop, Yahoo! Search). This may be a reasonable projection of where AI is headed. You can do a lot locally but there is too much incentive to centralize around cloud infrastructure, then the privacy concerns make that prohibitive and we end up with what feels like a false choice of cloud or bust.

Maybe it's LLM hype that will bring more powerful capabilities to the desktop?

I think classification and clusterisation is more important than search per se. "Where did I see that article about that weird psychological effect where people remember things they haven't seen" is not resolved by direct search, but can be - with some work - helped by language models and NLP. One doesn't need full blown frontier LLM for that, but bare word indexing would probably not do either.

Firefox is already using small local models for a few things, like tab grouping and link previews. I assume that this is for cases where a reasonably sized model isn't adequate.

> Firefox is already using small local models for a few things, like tab grouping and link previews

Which I immediately disabled alongside their "improved" Nova design, cause it doesn't improve productivity. These are all distractions from web browsing. If they want to improve anything, they should work on their abysmal spell checker, but that's not shiny enough.

Let's say the goal is doing something for work, so accuracy is important, and you prefer for it to be a nice reading experience, like a good translation.

As long as cloud model are somewhat better in these things, it's good that users would have the option to use cloud models.

Dont use this for work unless it has been verified by your corporate security and legal teams. Sharing business data with a company with which you dont have a enterprise agreement can land you in serious trouble.

Reply on news.ycombinator.com