Split brain experiments show that humans missing a functional corpus callosum largely operate in this way. I think it's an open question whether all humans operate in this way, or if communication between the hemispheres is necessary for 'real' explanations of behavior. The data we have at this point would support either conclusion.

You're right. Humans have short term memory, so we can genuinely recall our reasoning, but at the same time, once an action (or even a thought) has been had, we begin weaving a new narrative about why we did that thing or had that thought, that often has no resemblance to actual truth.

This is very reductive and gross misrepresentation of split brain experiments.

Humans have various level of memory (working, short term, long term), the ability to experience meta cognition, and the autonomy to make a decision - all of which (and much, much more) informs our behavior and are also things that an LLM fundamentally does not have.

These “studies” are painfully naive at best and are at worse an intentional misinformation campaign used to bolster the value of LLM past what they are useful for.