After all the fuss about lack of explainability in Deep Learning, this makes me believe the endgame will just be: politely ask the model to explain itself.
Then again, people do the same.
Since it's autocomplete, asking for the reasoning first makes it likely that the reasoning is used to calculate the answer. It would be interesting to see how accuracy changes when they are in the opposite order, and the reasoning is a post-hoc justification of the previously given answer.
You could also change the answer and see what the new post-hoc justification looks like.
"Why did you do that?" -> attempt to make up an answer that will be believable both to self-identity and to the listener.
So what's the point of the executive mind if the hindbrain is doing all the walking and talking?
The primary benefit of this story-making is that it allows formulating predictions and discarding the least probable. When a believable prediction kicks in, the predictive brain incentivizes the hindbrain with a partial taste of any reward or pain, with promises of more should the path transpire.
Disclaimer: not a neuroscientist, just a lifelong enthusiast of the subject.
Perhaps there is a similar route for AI models.