Comment by hkchad | Hacker News Reader

Comment by hkchad | Hacker News Reader

Parent Full thread

I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).

> It's free

It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.

> It's very possible that it costs you more than a cloud mode would

...which is almost always true in a single request/reply mode and never true in batch mode. Single request usually 2x-3x more expensive than cloud and batch mode 2x-3x cheaper. Now, for narrow tasks, a finetuned tiny 8b model would dramatically outperform SOTA frontiers for a fraction of price, esp. on energy efficient hardware like Apple.

Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting.

If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

> not 1 token on 1 session like local models.

Local models can absolutely run in batch, what are even talking about?

> If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.

> Local models can absolutely run in batch, what are even talking about?

I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.

If you buy the computer specifically for inference it is more expensive than cloud, but if you had it anyway it's free.

True, but how many people (realistically) buy a computer with 48GB+ of RAM?

48gb of vram.

a machine like this is about a years rent for most people.

a small car for most others.

I think he’s talking about the Mac Mini unified memory.

48G RAM is pretty useful if you want to run k8s locally for tests / exploration

true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires.

you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.

I actually have a Mac Mini M4 Pro with 48G. I gave the k8s example because this is what I was doing with it.

Was because I am back to using Linux as my workstation.

My Mac Mini is now a headless server for llama.cpp.

So, you are right that for these workloads , I would not be using the Mac Mini for k8s AND llama.

Another thing going against using a Mac for Linux containers is that there are no solutions that I know that properly manages memory : memory is given to the Linux vm , but never fluctuates if the needs in the vm are less than the initial request.

I know Orb Stack does that but is it proprietary. I think UTM does it , but not sure I would use UTM instead of Lima, Colima , multipass , etc to run containers.

Try smolvm microvms from https://smolmachines.com - among other benefits they only consume host resources if they're actually used.

It sounds like they are doing something similar to what I described in my other post below. Personal media station.

That can be done on hardware that quite a lot of people basically just have and don't use 24/7 to the max - because it is their gaming machine or their programming and compiling workhorse, for example. Of course you are paying for additional electricity but even with napkin-math instead of a "proper" calculation, you are unlikely to pay more for running your own instead of something commercial (and that can be offset further with some of the "modern" electricity contracts and/or PV and battery storage). Especially if we are talking about a stack that runs most of/all the time when you are not using your machine and makes LLM calls regularly while running.

The work in software/admin to get whatever you want set up is similiar no matter which infrastructure you use.

> the cost is included in your electricity bill

Acting like an extra $20 on my electric bill is equivalent to a $200/mo subscription is... a take.

Free heating in winter.

Though much less energy efficient than a heat pump.

Expensive heating but free computation then!

Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.

What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.

Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.

Reply on news.ycombinator.com