You absolutely could, and it could even end up just as accurate as human post-hoc rationalization. ;)

Self-analysis and self-interpretation is pretty clearly a key part of consciousness... I do wonder - how important to the process is the actual fidelity of the interpretation? Those people you meet who think they have deep insight into their own psyche while clearly having no clue... maybe they're p-zombies. ;)

That’s basically the gist of explainable AI