> This is an excellent use case for completely local, small model inference
Is it really ? LLM take lot of ram and drain battery. People run Firefox on low end computer.
Is it really ? LLM take lot of ram and drain battery. People run Firefox on low end computer.
On a 8GB system with a SSD it might be a bit slow due to having to keep experts on disk, but perfectly usable. On fast systems it may well be faster than the network round trip for many tasks.
What specific LLM and how much RAM does it take up to load? How does it run on somebody's 8GB RAM $600 laptop they bought from Best Buy? What pp/s and token/s rate on that hardware?
How much ram has a fair amount of flexibility since MoE can be kept on flash and swapped in at a performance cost, and depending on how small a context you can use.
Beside its mentioned 5 or 6 gb of ram.
People run Firefox on pentium with less than 8gb of ram, just 4 or 5 years ago 4gb computer were still extremely common. Even Microsoft sold its surface devices with 4gb of ram, I was dumb enough to buy one.
To give a counterpoint, you could disable it on those low end devices.
But then people would complain about the size, like what happened with Google Chrome embedded llm.
I think there wasn’t any bad choice, mistral was the safest.
Though low memory devices like 4-8gb probably does need specialized inference to give fair numbers.
> you could disable it on those low end devices
it's not like it does anything if you don't ask for it-- and good thing, because the privacy invasion would be all the worse if it did!
> mistral was the safest
Sending the user's confidential information to third parties while falsely suggesting that it is private can cause them irrecoverable harm.
An alternative is that a feature is slow for some users on slow hardware, and there is a setting to make it faster at the expense of privacy and security that they can switch on. The large body of users that find the default config fast enough have no reason to flip the switch.
> But then people would complain about the size, like what happened with Google Chrome embedded llm.
People or astroturf accounts? :P but having read some of the commentary on it there was some pretty weird takes, like generalized complaints about AI and people thinking google was using their computer to serve other people. Google's business model generally prevents correctly marketing the functionality in any case: it's not like they're going to make a proper pitch for how important it is for your privacy when the rest of their business is centered on hoovering everything up. Mozilla is not so constrained :P