Hacker News Reader

> A Mac Mini can run one LLM inference thread at a time.

That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

View on HN

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines.

Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in a datacenter serving 1,000 copies of myself? That's going to be more efficient.

Oh certainly, a local machine can't parallelize like the cloud can. Does anyone think that, though? What it can do is provide good-enough inference to support common business needs, without creating unnecessary data risks.

Why on earth would you need to do that for most things? Not everyone wants to be the next Coreweave. It sounds like you're killing a fly with a cannon to me.

Reply on news.ycombinator.com