Prtty much echoing what you said, in this video Cecila McGough, who has schizophrenia, talks about how important it is that people don't do anything to confirm her hallucinations.
Turns out the reason was that she saw an hallucination in that direction. So she looked away, she explains, in a follow up video 5 years later: (I found via the YouTube comments section)
https://m.youtube.com/watch?v=n7Wzb6esnpU
Here she's much better, and explains how she got better (in the beginning of the video)
What she describes is exactly what would happen with me (looking in the direction of the hallucination, together with the whole room).