"In fact" is a turn of phrase, it does not literally mean "in factuality."
Contextual fictional facts are really common in language. I remember doing it in philosophy first year. It conundrum-y philosophically, but usually understood effortlessly in normal discourse.