As for what happens to your body or anything else in the world? You can't ever get rid of those doubts. The only thing you can know without a doubt is that a doubter exists. Cogito ergo sum, as Descartes wrote.
Now, you could contrive a thought experiment where you clone me in such a way that neither I nor the clone know which one of us is the clone, but then, how is that any different in principle from babies switched at birth in a hospital?
Your memories don't define you, since they may be false. The only thing that defines you is your consciousness in the present. Everything else is I/O, and subject to interpretation and trust.
In fact, we do know that human memory is spectacularly unreliable, and, moreover, that it is a virtuoso liar, very accomplished at convincing us that we vividly remember things that never actually happened. It's proficient at rearranging memories and sensory impressions into ad hoc stories that we uncritically accept as true records of events.
From this point of view it seems a little over optimistic to me to say that we know ourselves, or to say that I know that the me of today is "the same person" (whatever I might mean by that) as the me of yesterday or last week or 1965.