You can rephrase that to an assumption that maybe everything other than those entities being simulated has no existence at all other than the bare minimum required for the observation. Whether or not the person you're responding to then exists at all as a subject of the simulation or are just another lazily instantiated entity that only exists to the point you interact with them is then an open question.

In other words, maybe you're the only subject. Or maybe I am. Or maybe all that exists are isolated "snapshots" and that we only assume time passes because we process a single momentary "image" of memories that includes the sensation of it passing.

The problem, of course, is that we can not possible ever know. And so we mostly discard this notion because absent someone pulling us through a door and letting us observe it from the outside, we're stuck inside a room and trying to infer the outside shape of the building without having any idea how much of the interior we have access to.