"we often make changes intended to improve the efficiency and throughput of our models.." I thought Anthropic said they never mess with their models like this? Now they do it often?
"we often make changes intended to improve the efficiency and throughput of our models.." I thought Anthropic said they never mess with their models like this? Now they do it often?
Never seen or heard of (from people running services at scale, not just rumours) this kind of API behaviour change for a the same model from OpenAI and Google. Gemini 2.5 Pro did materially change at time of prod release despite them claiming they had simply "promoted the final preview endpoint to GA", but in that case you can give them the benefit of it being technically a new endpoint. Still lying, but less severe.
The only explanations were either internal system prompt changes, or updating the actual model. Since the only sharply different evals were those expecting 2.5k+ token outputs with all short ones remaining the same, and the consistency of the change was effectively 100%, it's unlikely to have been a stealth model update, though not impossible.
I think that is compatible with making "changes intended to improve the efficiency and throughput of our models" - i.e. optimizing their inference stack, but only if they do so in a way that doesn't affect model output quality.
Clearly they've not managed to do that recently, but they are at least treating these problems as bugs and rolling out fixes for them.
>We've read and heard that you'd appreciate more transparency as to when changes, if any, are made. We've also heard feedback that some users are finding Claude's responses are less helpful than usual. Our initial investigation does not show any widespread issues. *We'd also like to confirm that we've made no changes to the 3.5 Sonnet model or inference pipeline.*That statement aged poorly. The recent incident report admits they "often" ship optimizations that affect "efficiency and throughput." Whether those tweaks touch weights, tensor-parallel layout, sampling or just CUDA kernels is academic to paying users: when downstream quality drops, we eat the support tickets and brand damage.
We don't need philosophical nuance about what counts as a "model change." We need a change log: timestamped, versioned, and machine-readable that covers any modification that can shift outputs: weights, inference config, system prompt, temperature, top-P, KV-cache size, rollout percentage, the lot. If your internal evals caught nothing but users did, give us the diff and let us run our own regression tests.
History proves inference changes can drastically alter outputs. When gpt-oss launched, providers using identical weights delivered wildly different qualities due to inference configurations.
We need transparency about all changes whether model weights or infrastructure. Anthropic's eval suite clearly missed this real-world regression. Proactive change notifications would let us run our own evals to prevent failures. Without this, we're forced to reactively troubleshoot. An unacceptable risk for production systems.