> Let's try a different approach.
“Let’s try a different approach” always makes me nervous with Claude too. It usually happens when something critical prevents the task being possible, and the correct response would be to stop and tell me the problem. But instead, Claude goes into paperclip mode making sure the task gets done no matter what.
It's mind-blowing it happens so often.
They will do ANYTHING but tell the client they don't know what to do.
Mocking the tests so far they're only testing the mocks? Yep!
Rewriting the whole crap to do something different, but it compiles? Great!
Stopping and actually saying "I can't solve this, please give more instructions"? NEVER!
Humans who are good at reasoning tend to ”feel” the amount of shaky assumptions they’ve made and then after some steps it becomes ridiculous because the certainty converges towards 0.
You could train them to stop early but that’s not the desired outcome. You want to stop only after making too many guesses, which is only possible if you know when you’re guessing.
(And imagine a CTO getting a demo of ChatGPT etc and being told "no, you're wrong". C suite don't usually like hearing that! They love sycophants)
(Using Claude sonnet with vscode where it consistently has issues reading output from terminal commands it executes)
It does seem to constantly forget that is not Windows nor Ubuntu it's running on
So far, at least, that seems to help.
Just take a look at zen-mcp to see what you can achieve with proper prompting and workflow management.
The current behavior amounts to something like "attempt to complete the task at all costs," which is unlikely to provide good results, and in practice, often doesn't.
I feel like we need a new base model where the next token prodiction itself is dynamical and RL based to be able to handle this issue properly
If it's true that models can be prevented from spiraling into dead ends with "proper prompting" as the comment above claimed, then it's also true that this can be addressed earlier in the process.
As it stands, this behavior isn't likely to be useful for any normal user, and it's certainly a blocker to "agentic" use.
The model should genwralize and understand when its reached a road block in its higher level goal. The fact that it needs a uuman to decide that for it means it wont be able to do that on its own. This is critical for the software engineer tasks we are expecting agentic models to do
No! The intern needs to actually understand what they are doing. It is not just one more sentence "by the way, if this fails, check ...", because you can never enumerate all the possible situations (and you shouldn't even try), but instead you need to figure out why as soon as possible.