How can we even be sure we are the same person as yesterday? Or even five minutes ago.. It's all pretty weird.
Conveniently, this makes death less scary.
Further reading: Zen and the Art of Consciousness by Susan Blackmore
It might depend on how you define "person". When I look around, I see distinct human bodies. When I wake up, I'm in the same body as when I went to sleep, so in that sense I'm the same person.
But if memories are just physically encoded in the brain, I don't think this scenario makes sense even as a thought experiment.
When "I" wake up in the other person's body, I would have all their memories and none of my previous memories. So I wouldn't even know.