If, however, you are making an objective claim about "past lives" and reincarnation - there is no solid evidence for it. 99% of "past life memories" examples are in the form of "I had a family and we had a house" (omg, that sounds like our dead relative, you must be him reincarnated), or in a form of an obviously imagined past life of a historic figure, with details which have been read in history books by the individual claiming to experience the said past life. And interestingly they always seem to read about the historic figure first (giving their subconscious mind the change to soak up that knowledge (in this life though)), and "experiencing a past life" only after.
So to answer your question - because you only get one particular mind, and that's it. This is what science shows us. Whether you "feel like" you are "one with all other minds" is just a subjective experience. It's not physical reality.
But them I'm not sure we'd still be "us", anymore, by that point.