Maybe I'm the only one who is real, and everyone else are just robots. I'm not sure I'll ever be able to know for sure.
Conveniently, this makes death less scary.
Further reading: Zen and the Art of Consciousness by Susan Blackmore
It might depend on how you define "person". When I look around, I see distinct human bodies. When I wake up, I'm in the same body as when I went to sleep, so in that sense I'm the same person.
But if memories are just physically encoded in the brain, I don't think this scenario makes sense even as a thought experiment.
When "I" wake up in the other person's body, I would have all their memories and none of my previous memories. So I wouldn't even know.