Hacker News Reader

Comment by millicentricism | Hacker News Reader

With roughly 2.7 million seconds per month, times 14 tokens per second, you are getting 38.5 million tokens a month at most.

That’s less than 164USD worth of GLM5.3 tokens on the inference market. So that 6000 USD rig will take 3 years to break even - and only if it runs continuously. And this is being generous, as it’s not even taking quantisation into account.

View on HN

I think the “killer app” is doing inference without sending the data to China or the US. At home it’s overkill but imagine you are an EU consultancy with a lot of client data to work on, or a company/institution with a lot of sensitive data, buying the hardware to make sure the data stays private is a huge benefit. So is that you “own” the model. Its capabilities, price or access don’t change at someone else’s whim.

Some of that is that EU providers need to up their game here.

Needing an EU native option is really the one and only reasonably objection I've heard against using LLMs from the cloud, the rest is tin-foil hat level unless you're actually intending to meddle with the inference or fine tuning or something beyond just querying.

I think if you steel-man what I'm saying, what you're saying falls apart. 14 tokens per second was rare. It only dropped that low in one scenario where he had it single shot an entire game (flappy bird clone) from scratch, with different assets, all self created, and so on. It ended up resulting in the LLM doing stuff like plotting out a some odd 100 item long to-do list, requerying it repeatedly, and so on. And it succeeded.

Also as the video mentions, the guy wasn't very familiar with what he was doing, and so there are almost certainly various optimizations on the config side he could work out, especially as he was using a 5 GPU system, which default configs are probably not well optimized for.

But I think we've rapidly moving along the same path as image gen stuff. Local generation has gone from purely theoretic, to requiring supercomputers to run relatively incapable models, to where we are today - where with a fairly basic high end setup, he's comfortably running a frontier level model. There's definitely an argument for going local that's only growing stronger by the day.

I agree it’s probably not representative token speed. But I do believe the overall observation holds: The monetary value of local inference is bound by the wall clock.

I agree that there are many other reasons than cost alone.

> That’s less than 164USD worth of GLM5.3 tokens on the inference market.

I can cherry pick stats too.

The other day I heard mention of someone paying $200/mo for Claude Code.

At those rates my local LM setup pays for itself in a single year.

Reply on news.ycombinator.com