Tangent: This is often true of humans as well.
We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.
We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.
When you ask people who are rationalising poor behaviour about the scenario, but it is someone else doing it, they may arrive at a better answer. Can we use multiple LLMs to achieve self criticism and critical thinking?
To your point, I agree that nominally there should be a way to give conceptual names to paths of weights, and when answering a question, notice which weights were and were not applied and retrospect on that.
That's not what reasoning traces as they currently exist are, though.
Is it unreasonable to hold LLMs to a higher standard?
I disagree. In fact, I find people who think of themselves as highly rational seem to be particularly prone to rationalization. Because being rational is integral to their whole identity, they are much more eager to accept their own rationalizations as sound and resist admitting that they are guided by instinct.