Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And we're in the middle of a massive gold rush. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law mean nothing to them. It's just something you pay off in a settlement if you get caught, a cost of doing business.
With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap.
For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant. This way I get a lot closer to SOTA-model retrieval quality (like with Office Copilot 365 looking for conversations in Teams).
PS: I wouldn't be surprised if Microsoft runs something similar on their end :)
But yes the Mac Studio is outrageously priced especially for running such a small model as qwen 27b. For that price you can use something much much cheaper. It only shines for models that are much bigger, because there simply is not much hardware that can address fast 512GB banks.
Which is a regulation constraint on many professions, BTW. Some people simply can't give some data to say, ChatGPT, without comprehensive guarantees written in Sam Altman's blood.
There need to be second order consequences to those who delegate their responsibility to companies that behave like this. They know the contract is worthless, but then proceed to use it as defense for their own gross negligence.
They ensure you are not liable when they misuse the information you entrusted them because you took “adequate precautions”.
It was pretty easy mode like this tbh, though you do need to know what you're doing. I use it with llama-server and openwebui.
I guess you can get it a lot more click to go with something like LM Studio though but I want to use it on a server and call upon its services from multiple sources.
I just run a server with Linux (previously multiple servers but I found a way to add multiple GPUs to a single one).
> For that price you can use something much much cheaper.
I meant "For that model size you can use something much cheaper".
This is a huge motivation for me to build one myself
The new M5 Ultra should deliver ~50% faster token generation (1.2TB/s mem bandwidth); and extrapolating from my M5 Max (since the M5 Ultra is literally just 2x Maxes), probably ~3x faster PP.
But I don't think it's fair to look at this only from monetary ROI vs API. With local models, you get privacy and ownership.
I do not trust _any_ API provider with my most personal information; such as for example, all my messages, emails, daily journals spanning a decade+, all my photos and videos, etc. So it unlocks new use cases that I simply don't feel comfortable with via API.
And a personal assistant with ALL my context and data, locally, has been incredibly useful for me :) Zero outages either, zero "overloaded", etc. Nearly-zero refusals too (I don't run abliterated models; thinking prefill has worked for anything I've wanted to do)
I realize the price of NVIDIA has gone up but there are plenty of GPU options from others like AMD and Intel with reasonable performance.