Hacker News Reader

Comment by Razengan | Hacker News Reader

Theory (Conjecture? Hypothesis?): What we notice as "model nerfing" is the company diverting compute to training/running new unreleased models..

Remember that some people get access to the next flagship version long before us peasants do. I recall seeing the mention of "Astra" more than a month before it was officially announced

View on HN

wouldn't less compute result in slower inference, rather than worse performance?

They could potentially quantize the model and run it at lower quality taking less VRAM.

The more likely thing that would happen is that the provider begins silently interpreting (perhaps some) high effort-level requests as medium, etc., or having a classifier do this far more subtly. As such, the load on the cluster is less, and more resources can be devoted to training. Whether the frontier labs actually do this is purely conjecture at this point.

I assume there's classification going on where a really basic "Hi how are you?" style request sent to a high-effort instance can be routed to a lower-level instance. This... is pretty much fine with me, assuming they do a good job of it.

I would also assume they use nebulous labels like "Medium Effort" or "High Effort" map to quantitative amounts of compute allocation... and that these amounts can be varied manually or automatically. Right?

I mean, there's a reason why they call it "High Effort" and not "Exactly 5 Minutes of GPU Time on Exactly 10 GPUs." They want to be able to move those sliders and tweak those knobs.

The problem is that if benchmarks are run at a tight classifier that says "a request for high effort means check-under-every-stone regardless of simplicity" but a user request is run on a different classifier where "high means maybe high, maybe medium, maybe even low, even for meaningful tasks" then you're not getting the model that you saw in the benchmarks.

And, while you might be billed fewer tokens as a result (because the lower thinking would result in less investigatory work), you might not know this is happening, and know to dial up effort accordingly - you'd simply get a worse work product. And certainly, Anthropic's incentive for anyone on a subscription is to push this as aggressively as they can, so people use less of that subscription.

Sadly, I'd also expect that the OP's benchmark will be detected as a test of model capabilities, and thus be given a high classification so that this strategy remains undetected.

why is that more likely?

Because they already did so. The model in Codex will get lower `juice` than API version.

My guess is that they are dynamically changing the quality of the model to always keep the speed above some floor. So once it gets below that they switch to a worse quant or reduce reasoning level, or some combination of both.

There is no nerfing, look at the data before coming up with a theory as to why the nerfing that isn't even happening is happening.

fuck

Reply on news.ycombinator.com