Comment by stkdump | Hacker News Reader

Comment by stkdump | Hacker News Reader

Parent Full thread

I have a computer with a 5090 on a smart plug at home running Qwen3.8 27B for agentic coding. On a busy day it can use around 5kWh, though on most days it is around 2kWh. I am sure cloud is more efficient because there is probably efficiency in running many parallel streams, some of the models have fever than 27B active parameters and the power draw of the non-GPU components is also spread over more GPUs. But still I also believe that the energy use numbers you get from the inference providers are a bit "beautified". After all, they still fight a political battle and have to show that it isn't all so bad.

Having said that, seeing the incredible progress of models throughout this year, I also strongly believe that the planned buildout is overeager. Even I with my gaming hardware often run out of instructions to give. And the smarter the models get that I can run, the less I will be able to saturate my hardware. Is it because of my lack of creativity of which kinds of tasks I can give to AI? Maybe a bit, but currently I can't believe that I am that far away.

I suspect that some/a large portion of the build out is due to OpenAI/Anthropic/etc. just throwing hardware at the problem -- why bother trying to make training and inference efficient when you can just throw hardware/money at the problem. I remember NVIDIA talking about their supercomputer clusters with a large number of interconnected GPUs.

I think the power savings and efficiencies (that the large companies have also benefited from) have come from 2 areas:

1. open source and local AI enthusiasts -- think things like llama.cpp, quantization, etc.

2. Chinese labs and other smaller/research companies like Mistral that are using constrained hardware -- see the various advancements in the various models to reduce compute complexity such as mixture of experts [1], sharing key/value data between a group of layers, etc.

[1] Though the original idea for mixture of experts comes from a 1991 research paper (https://huggingface.co/blog/moe), so maybe a third area is research from Universities, etc.

The incentives are rather in the opposite direction. Enthusiasts are trying to get models to run at all on the hardware they have, even if the resulting efficiency is poor, whereas a big company spending billions on hardware can afford to hire hundreds of performance engineers to tune their systems end-to-end for maximum efficiency, since the expense pays for itself even if they only manage to eke out a 1% improvement. I.e. why not make training and inference efficient when you can just throw money at the problem?

Because at the moment they have basically infinite money and it's almost always faster to throw hardware at the problem than optimizing the solution, given infinite money.

Given infinite money, after buying up the finite amount of hardware available, you'll still have infinite money left to optimize it for maximum efficiency. And at any finite budget, both the optimal amount to spend on hardware and the optimal amount to spend on efficiency grow with the total budget available, although their relative shares may vary. So while an enthusiast may refuse to invest in hardware at all and focus 100% on efficiency, their efforts will still be outclassed by a large company allocating 99% to hardware and 1% to efficiency.

That's simply untrue, they do not have infinite money nor infinite hardware. They have far stricter limits on hardware than they do on development effort to get more out of their finite hardware.

They are acting like they have infinite money (huge annual losses [1], [2], [3]) and infinite hardware (aiming to build a large number of data centres with a lot of GPU hardware [1], [3], [4]) even if that isn't the case.

[1] https://www.morningstar.com/stocks/anthropics-leaked-financi...

[2] https://www.wheresyoured.at/anthropics-profitability-swindle...

[3] https://fortune.com/2026/06/16/openai-financials-leaked-loss...

[4] https://www.forbes.com/sites/paulocarvao/2025/12/06/why-open...

This is such a hilarious comment when we've literally just had Sol 6 and Opus 5.5 drop that are massive wins for efficiency. I mean, how out of touch can you be? Especially if you think power saving and efficiency come from local AI enthusiasts. What a fucking joke.

> why bother trying to make training and inference efficient when you can just throw hardware/money at the problem.

Sure, for certain bleeding edge research path-finding, any optimization is premature but that's tiny compared to deployed end-user product scale. The biggest constraints frontier cloud model providers face are: 1. Getting enough of the hardware they want, 2. Capital, 3. Talent (and #3 is a largely a function of #2 & #1).

Given the raw economics, it would be foolish to not have efficiency teams fast-following their deployments as closely as possible. We're seeing direct and indirect evidence of significant efficiency improvements across all the frontier providers and the rate seems to be increasing. The coming OAI & ANT IPOs hinge on demonstrating they can serve many use cases beyond software dev profitably at what they are willing to pay, which is generally less than software dev. The trend is clearly "the higher volume the use case, the less they're willing to pay".

Inference vs training. It's training that takes enormous amount of power, both electrical and the compute. I suspect many many models get simply thrown away because they end up being too low on benchmarks by the time they are done. And some are not released to the public. So we learn only about tiny percentage of trained models and their environmental impact.

Hey guys, has anyone seen my goalposts?

More seriously, that needs to be done once and then it can be used by millions of people. Divide the cost by all the users and it's trivial. It's certainly not enough to lose sleep over.

Similarly with my Strix Halo running DSV4. Got a meter for it a couple of weeks back, and, during inference, it's pulling about 160W, and I run it about 15 hours a day on a busy day, or about 2.4kW. I'd say an average day is more like 2 hours for me. It's a small impact on my monthly bill. Owning a hot tub is orders of magnitude worse, in my experience.

Batching is extremely powerful for optimizing power use. Your card is spending the vast majority of it's energy use fetching weights from it's memory, and it only gets to use each weight once. Running that same model on a bigger machine with lots of batching gets to amortize the cost of loading a weight across all the parallel requests. N=128 is literally about 50 times more energy-efficient than N=1.

I don't know. It would surprise me. For long agentic work you shouldn't need that many streams for the KV cache to become larger than the (active) parameters. At 128 streams the model weights would be neglectable. I am using a 22GB model file, i.e. I have 10GB for context, which isn't even the max that the model supports.

Reply on news.ycombinator.com