"Use your web search tool to find me the go-to component for doing xyz in $language $framework. Always link the GitHub repo in your response."
Previously Sonnet 4 would return a good answer to this at least 80% of the time.
Now even Opus 4.1 with extended thinking frequently ignores my ask for it to use the search tool, which allows it to hallucinate a component in a library. Or maybe an entire repo.
It's gone backwards severely.
(If someone from Anthropic sees this, feel free to reach out for chat IDs/share links. I have dozens.)
Sonnet 3.5 did this last year a few times, it'd have days where it wasn't working properly, and sure enough, I'd jump online and see "Claude's been lobotomized again".
They also experiment with injecting hidden system prompts from time to time. Eg. if you ask for a story about some IP, it'll interrupt your prompt and remind the model not to infringe copyright. (We could see this via API with prompt engineering, adding a "!repeat" "debug prompt" that revealed it, though they seem to have patched that now.
> I started running my prompts through those, and Sonnet 3.7 comparing the results. Sonnet 3.7 is way better at everything.
Same here. And on API, the old Opus 3 is also unaffected (though that model is too old for coding).
IDK about you but I find it faster to type a few keywords and click the first result than to wait for "extended thinking" to warm up a cup of hot water only to ignore "your ask" (it's a "request," not an "ask," unless you're talking to a Product Manager with corporate brain damage) to search and then outputs bullshit.
I can only assume after you waste $0.10 asking Claude and reading the bullshit, you use normal search.
Truly revolutionary rechnology
Might be Claude optimizing for general use cases compared to code and that affecting the code side?
Feels strange, because Claude api isn’t the same as the web tool so I didn’t expect Claude code to be the same.
It might be a case of having to learn to read Claude best practice docs and keep up with them. Normally I’d have Claude read them itself and update an approach to use. Not sure that works as well anymore.