The real problem is locality. Why am I located in me instead of in you? Why is it so much easier for me to see out of my eyes than it is for me to see out of yours? What's the connection between my perspective and my body?
The idea that self-awareness/consciousness ("I am me") is the big problem is weird. I have no problem locating that in the material world. If I program a computer to poll all of its peripherals, report on its RAM and processors, and adjust how it runs its programs based on that, that's as much self-awareness as I have, if not more.