/on The prose is load-bearing unbearable — every sentence feels like it was engineered to sound profound rather than to be read.
I remember how I enjoyed agents between December and February, something started changing around March.
I thought models are getting dumber, but benchmarks were convincing opposite, initially I thought maybe they're quantizing models for day to day use, but Opus 4.8 and Opus 5 seems worse models than Opus 4.6
The economics catching up with the providers in regards to how much compute they can burn per request and have it make sense for them financially?
A sort of model collapse where Opus 5 seems to love throwing out long paragraphs of text and it needs to be "fixed" by changing the output style and other patches.
I'm not sure, it might also catch up to Kimi K3 and GLM 5.3 and the models that I'm moving to from Anthropic.
I think we just need to decouple "doing better on benchmarks" and "actually more useful to me", since they've clearly diverged
>Opus 4.8 and Opus 5 seems worse models than Opus 4.6
After all we've heard about benchmark cheating, I'm earnestly not sure which or whether benchmarks are reliable anymore. But, beyond the models, I wonder if changes to their harnesses and/or instructions dumb them down. I have noticed models change their behavior, even when using the same version/effort. Sometimes for better. Sometimes for worse.
And, I have noticed a model go from really good to struggling. On 4.8 things were going well for a good stretch, so I did not switch to 5 when it came out. Even after hearing complaints about 5, 4.8 was still going well. Then, suddenly over the last few days, 4.8 seems to have nosedived. It feels similar now to the complaints I hear about 5.
In my case, it suddenly started ignoring my design guide, and introducing new fonts etc. It would even use several different fonts and sizes, as well as different margins for similar elements within the same page. It abandoned classes and started inlining styles. It started feeling random and, even after it realized it needed to go back to the design guide, it just continued with more of the same.
There seems to be something that happens after new model releases in both quality and behavior of previous models. It may not be immediately, but eventually there is frequently some regression.
If so, I would think that would result in worsened performance, not quality (unless you are also suggesting they may be quantizing).
In addition to that, while yes, 4.6 and 5.0 can solve problems, they do so differently. Sometimes 5.0 does better by a wide margin but that would be expected as they are supposed to be better.
models are incentivized by their makers to burn through as many tokens as they possibly can, so long as the customer doesn't cancel.
Opus models have degraded rapidly since March, with each release being considerably less useful and considerably more verbose. The language is no so weirdly florid it’s difficult to understand, and its logical conclusions are almost always suspect. It goes off on clearly bizarre snipe hunts to the point it feels like I’m using a gpt 3 model at times. It’ll announce that it’s about to embark on building something then just return control to the user and wait. You can also tell perceptibly when they’re reducing model quality to load shed - it becomes stupider and stupider to the point you’re better off dumping state and switching to codex or just turning in for the day and hoping they secured more capacity tomorrow.
It’s an absolute race to the bottom with Anthropic on virtually every level. I’ve rarely seen a company so rapidly accumulate good will in the developer community as they did around 4.6 in December and January. By March, it was inconceivable to use anything else. 4.8 was a bit of a wake up call to not put all your harness eggs in one basket. 5 is straight up time to cancel territory.
I actually manually set my model back to the older versions to get anything serious done. More and more I use codex for anything non trivial.
This isn’t about avarice by the provide trying to get more tokens and more utilization. They’ve over subscribed for capacity as it is. If they can produce better quality for less tokens they can charge a higher margin and will be paid if, which is a better economic strategy overall. This is something else. I suspect it’s actually the opposite, they’re finding ways to cut capacity demand in ways that leads to worse behavior that leads to more capacity demands, worse output, worse quality, and worse margins, worse, worse, worse.
Just as I never saw a company accumulate such positive developer good will so fast, I’ve never seen one squander it so fast too.
It was the wrong time for the GP to drop that subscription from $200 to $20, because $200 gets you a metric assload of cognition while $20 gets you nothing beyond what a local model running on your own graphics card can deliver.
I'd guess you're deliberately exaggerating here, but still. I've never clocked the actual tokens/second, but I'm on the $20 plan and get ~15M tokens/month for fully utilized weekly quotas (checked couple months ago). Meanwhile the best I've been able to get locally was ~8 tokens/second with Qwen3.6 35B A3B, which is wildly painful for coding sessions and gets a maximum ~20M tokens in a month... if it's going 24/7.
Just wanted to stick some empirical data here, given that statement.
At $1500 that's 75 months of $20/mo Claude which are MUCH better models than you can run locally.
The point raised by this very article is that you can't depend on that. It's Flowers for Algernon As A Service.
One challenge I run into is I run many agents at once, so local resource are a tiny fraction of the total inference we’re using. Every developer with his own Pro 20x, Max, etc. accounts is letting us hit literally trillions of tokens a month.
Consistent, predictable behavior is valuable, even more so given the nondeterministic nature of LLMs. Nondeterminism combined with unpredictability might as well be randomness.
I was a 4.6 acolyte from April til the fable drop, lost that quick, cancelled and took a break, came back a month later, tried opus 5 and liked it, so unpinned 4.6.
Results were great at first, and they're still not terrible, but I have noticed a regression in accuracy, so to speak, where I am pointing out issues that are quite obvious in review.
I pretty much use sonnet 5 low/medium when I have a plan to solve a simple problem and depending on scope, opus low/medium for more complex/bigger scope implementation, and only go high when it's very complex or I'm spitballing architecture/solutions and iterating plan. Never go xhigh or max.
The verbosity is insane though, opus 5 documents everything and just regurgitates whatever lead it to the design choice in there, which makes it more opaque because it's talking about something that was discussed once in a session that no one else can see (except their backend ofc)
I don't even try to steer it away from that with harness, because it doesn't work and just ends up agonizing over whether it should write some comment. Three paragraphs waffling on that on verbose output
I did however have it write a script that basically is git add -A -p for comments though, haha.
I'm $20/month, have all my telemetry toggles off, don't really over engineer prompt/context, just some basic skills for repeated patterns.
If I give it the specific instruction to "elide all revisions, corrections, and past mistakes" it usually works. You can also have Sonnet do a cleanup writing/style pass in a subagent. I impression is that Opus has been deliberately trained to keep track of all such revisions by default as a kind of ad-hoc memory mechanism. It's probably good for autonomous coding and beating benchmarks, and I presume reduces flailing when a separate session needs to pick up the work.
it really angers me every day