"Why of course, sir, we should absolutely be trying to compile python to assembly in order to run our tests. Why didn't I think of that? I'll redesign our testing strategy immediately."
I would imagine this all comes from fine tuning, or RLHF, whatever is used.
I’d bet LLMs trained on the internet without the final “tweaking” steps would roast most of my questions … which is exactly what I want when I’m wrong without realizing it.
Not always. The other day I described the architecture of a file upload feature I have on my website. I then told Claude that I want to change it. The response stunned me: it said "actually, the current architecture is the most common method, and it has these strengths over the other [also well-known] method you're describing..."
The question I asked it wasn't "explain the pros and cons of each approach" or even "should I change it". I had more or less made my decision and was just providing Claude with context. I really didn't expect a "what you have is the better way" type of answer.
Seems to help.
- “I need you to be my red team”(works really well with, Claude seems to understand the term)
“analyze the plan and highlight any weaknesses, counter arguments and blind spots critically review”
> you can't just say "disagree with me", you have to prompt it into adding a "counter check".