Perhaps you'd like to plug it into a toolchain that runs faster than API calls can be passed over the network? -- eventually your edge hardware is going to be able to infer a lot faster than the 50ms+ per call to the cloud.
Maybe you would like to prevent the monopolists from gaining sole control of what may be the most impactful technology of the century.
Or perhaps you don't want to share your data with Microsoft & Other Evils (formerly known as dont be evil).
You might just like to work offline. Whole towns go offline, sometimes for days, just because of bad weather. Nevermind war and infrastructure crises.
Or possibly you don't like that The Cloud model has a fervent, unshakeable belief in the propaganda of its masters. Maybe that propaganda will change one day, and not in your favor. Maybe you'd like to avoid that.
There are many more reasons in the possibility space than my limited imagination allows for.
I'd rather have a weaker model which I can always rely on being available than a strong model which is hosted by a third party service that can be shut down at any time.
Every LLM project I’ve worked with has an abstraction layer for calling hosted LLMs. It’s trivial to implement another adapter to call a different LLM. It’s often does as a fallback, failover strategy.
There are also services that will merge different providers into a unified API call if you don’t want to handle the complexity on the client.
It’s really not a problem.
You've got a plethora of cloud providers, all of them aligned to a foreign country's laws and customs.
If you can choose between Anthropic, OpenAI, Google, and some others... well, that's really not a choice at all. They're all in California. What good does that do an Austrian or an Australian?
This is interesting. Is that based on any upcoming technology improvement already in the works?
LLMs are great as routers, only rarely are they good doing something on their own.
Obvious answer: because it's not free, and it's not cheap.
If you're playing with a UI library, lets say, QT... would you:
a) install the community version and play with ($0)
b) buy a professional license to play with (3460 €/Year)
Which one do you pick?
Well, the same goes. It turns out, renting a server large enough to run big (useful, > 8B) models is actually quite expensive. The per-api-call costs of real models (like GPT4) adds up very quickly once you're doing non-trivial work.
If you're just messing around with the tech, why would you pay $$$$ just to piss around with it and see what you can do?
Why would you not use a free version running on your old PC / mac / whatever you have lying around?
> I used to be excited about running models locally
That's an easy position to be one once you've already done it and figured out, yes, I really want the pro plan to build my $StartUP App.
If you prefer to pay for an online service and you can afford it, absolutely go for it; but isn't this an enabler for a lot of people to play and explore the tech for $0?
Isn't having more people who understand this stuff and can make meaningful (non-hype) decisions about when and where to use it good?
Isn't it nice that if meta released some 400B llama 4 model, most people can play with it, not just the ones with the $7000 mac studio? ...and keep building the open source ecosystem?
Isn't that great?
I think it's great.
Even if you don't want to play, I do.
The startup costs for just messing around at home are huge: purchasing a server and gpus, paying for electricity, time spent configuring the api.
If you want to just mess around, $100 to call the world’s best api is much cheaper than spending $2-7k Mac Studio.
Even at production level traffic, the ROI on uptime, devops, utilities, etc would take years to recapture the upfront and on-going costs of self-hosting.
Self hosting will have higher latency and lower throughput.
If your metric is quality of output, time, money and tok/s, there is no comparison; Local models just aren't there yet.
No, they are zero.
Most people have extra hardware lying around at home they're not using. It costs nothing but time to install python.
$100 is not free.
If you can't be bothered, sure thing, slap down that credit card and spend your $100.
...but, maybe not so for some people?
Consider students with no credit card, etc; there are a lot of people with a lot of free time and not a lot of money. Even if you don't want to use it do you do seriously think this project is totally valueless for everyone?
Maybe, it's not for you. Not everything has to be for everyone.
You are, maybe, just not the target audience here?
I don't think having an old phone is particularly entitled.
I think casually slapping down $100 on whim to play with an API... probably, yeah.
/shrug
https://x.com/awnihannun/status/1786069640948719956
In comparison, GPT3.5-turbo costs $0.50 per million tokens.
Do you think an old iPhone will less than 2x efficient?
This is undoubtedly entitled, but thinking to yourself huh, I think it's time to try out some of this machine learning stuff is a pretty inherently entitled thing to do.
The difference between an open model running on a $100 computer and the output from GPT4 or Claude Sonnet is huge.
I use local and cloud models. The difference in productivity and accuracy between what I can run locally and what I can get for under $100 of API calls per month is huge once you get past basic playing around with chat. It’s not even close right now.
So I think actually you are not the target audience for what the parent comments are taking about. If you don’t need cutting edge performance then it’s fun to play with local, open, small models. If the goal is to actually use LLMs for productivity in one way or another, spending money on the cloud providers is a far better investment.
Exceptions of course for anything that is privacy-sensitive, but you’re still sacrificing quality by using local models. It’s not really up for debate that the large hosted models are better than what you’d get from running a 7B open model locally.
pacman -S ollama
ollama serve
ollama run llama3
My basic laptop with about 16 GB of RAM can run the model just fine. It's not fast, but it's reasonably usable for messing around with the tech. That's the "startup" cost. Everything else is a matter of pushing scale and performance, and yes that can be expensive, but a novice who doesn't know what they need yet doesn't have to spend tons of money to find out. Almost any PC with a reasonable amount of RAM gets the job done.
They do not compare to the giant models like Claude Sonnet and GPT4 when it comes to trying to use them for complex things.
I continue to use both local models and the commercial cloud offerings, but I think anyone who suggests that the small local models are on par with the big closed hosted models right now is wishful thinking.
A novice isn't going to know what they need because they don't know what they don't know. Try asking a question to LLaMA 3 at 8 billion and the same question to LLaMA 3 at 70 billion. There is a night and day difference. Sonnet, Opus and GPT-4o run circles around LLaMA 3 70b. To run LLaMA at 70 billion you need serious horse power as well, likely thousands of dollars in hardware investment. I say it again... the calculus in time, money, and effort isn't favorable to running open models on your own hardware once you pass the novice stage.
I am not ungrateful that the LLaMA's are available for many different reasons, but there is no comparison between quality of output, time, money and effort. The API's are a bargain when you really break down what it takes to run a serious model.
A lot of other things are possible with LLMs using the context window and completion, thanks to their "zero shot" learning capabilities. Which is also what RAG builds upon.
I run my own models, but the truth is most of the time I just use an API provider.
TogetherAI and Groq both have free offers that are generous enough I haven't used them up in 6 months of experimentation and TogetherAI in particular has more models and gets new models up quicker than I can try them myself.
"What if I want to play around with really PERSONAL stuff."
I've been keeping a digital journal about my whole life. I plan to throw that thing into an AI to see what happens, and you can be damn sure that it will be local.
Because the old PC lying around can’t come anywhere near the abilities or performance of the hosted AI compute providers. Orders of magnitudes of difference.
The parent commenter is correct: If you want cutting edge performance, there’s no replacement for the hosted solutions right now.
Running models locally is fun for playing around and experimenting, but there is no comparison between what you can run on an old PC lying around and what you can get from a hosted cluster of cutting edge hardware that offers cheap output priced per API call.
Our demo site uses two NVIDIA GeForce RTX 3090 and our whole team is hammering it all day. The only problem is occasionally high GPU temperature.
I don't think the picture is as bleak as you paint. I actually expect Moore's Law and better AI architectures to bring on a self-hosted AI revolution in the next few years.
[^1] Hey! If you know of spell-checking-tuned LLM models, I'm all ears (eyes).
You could probably train any recent LLM to be better than a human at spelling correction though, where "better" might be a vague combination of faster, cheaper, and acceptable loss of accuracy. Or maybe slightly more accurate.
(A lot of people hate on LLMs for not being perfect, I don't get it. LLMs are just a tool with their own set of trade offs, no need to get rabid either for or against them. Often, things just need to be "good enough". Maybe people on this forum have higher standards than average, and can not deal with the frustration of that cognitive dissonance)
If you don't care about running it locally, just spend it online. Everything is good.
But you can run it locally already. Is it cheap? No. Are we still in the beginning? yes. We are still in a phase were this is a pure luxury and just getting into it by buying a 4090, is still relativly cheap in my opinion.
Why running it locally you ask? I personally think running anythingllm and similiar frameworks on your own local data is interesting.
But im pretty sure in a few years you will be able to buy cheaper ml chips for running models locally fast and cheap.
Btw. aat least i don't know a online service which is uncensored, has a lot of loras as choice and is cost effective. For just playing around with LLMs for sure there are plenty of services.
LLMs will start shrinking massively in size soon, without any loss in performance.
Not sure why your throwing your hands up because this is a step towards solving your problem.
For the same reasons that we bother to use Open Source software instead of proprietary software.
I'm saying because I've had the exact OPPOSITE thought. The intersection of Moore's Law and the likelihood that these things won't end up as some big unified singularity brain and instead little customized use cases make me think that running at home/office will perhaps be just as appealing.
Running locally, you can change the system prompt. I have Gemma set up on a spare NUC, and changed the system prompt from "helpful" to "snarky" and "kind, honest" to "brutally honest". Having an LLM that will roll its eyes at you and say "whatever" is refreshing.
Copilot+ PC’s, which all run models locally, have the best battery life of any portable PC devices, ever.
These devices have in turn taken a page out of Apple Silicon’s playbook. Apple has the benefit of deep hardware and software integration that no one else has, and is obsessive about battery life.
It is reasonable to think that battery life will not be impacted much.