Hacker News Reader

Comment by jack_pp | Hacker News Reader

This framing local LLMs as free is stupid. Basically pay 100+ months worth of API costs up front isn't free in the slightest. And it will be slower than non-local, your hardware will be outdated in 12 months and probably won't be able to run SOTA at anywhere near non-local speed in max 20 months

View on HN

Yeah, it glosses over a gigantic capital expenditure. It's sort of like saying that an open source modern CPU architecture allows you to build your own CPU "for free" (provided that you own and operate a fab).

True. But there are other meanings of "free". I.e. nobody can say "from now on you no longer have access to model X because you're an asshole"

Some obvious examples of why you'd want to spend the capital on this would be, for example, making some kind of autonomous system which needs to be periodically be offline, or you need complete confidentiality of what you're using the model for, etc.

To be cost effective with inference providers, you have to find some way to be using it 24/7.

The ecosystem for inference is centralized around a few core projects, i.e. vLLM, sglang, and llamacpp.

If they decided to collude, they could absolutely say "from now on you no longer have access to model X because you're an asshole"

The commercial inference offering are also downstream of one of those 3 projects (or trt-LLM if they're nvidia). It would impact Ollama, and fireworks, together, and everyone else.

Don't tempt fate.

and I think asking for a whole dissertation on the hardware demands every single time is stupid. the point is that even people and organizations with capital couldn't do this before either.

if it doesn't apply to you then just come back in a couple years and see what the situation is then. 1 million context window, 1 million tiny layers to fit in 4gb RAM at a time, with 256gb of fast unified RAM in every consumer device? Or a different concept entirely

in the meantime, z.ai probably doesn't reply to US subpoenas so you can shift all your incriminating conversations over to that and use GLM anyway. who cares if the Party trains on your data and steals your IP and ignores you for legal matters, when the alternative in the US is just a thin corporate layer and party who steals your IP and will snitch on you for legal matters.

Hardware outdated in 12 months is FUD. What that would mean in practice would be either affordable consumer GPUs with > 32GB of VRAM, which doesn't look like it's going to happen, or unified memory systems with much higher bandwidth. That also seems unlikely.

You're better off setting a budget and buying the best machine you can afford in that range, or picking a VRAM target and accepting the class of models you can run on it. Those models will almost certainly improve over time and your skills will adapt to the limitations. Hardware is so valuable right now that it's not even likely to be a significant loss if you had to sell.

Right now I think 24 GB is probably the best bang for your buck (used 3090), because you also get a high end gaming/gpgpu device which is nice anyway. 32 GB you can do with AMD or Intel, but NVIDIA is megabucks and at this point you're really paying for RAM. Unfortunately the ship has sailed on "reasonably" priced RTX 6000s, which at one point were about $7k and are being listed at $10k++.

Reply on news.ycombinator.com