But in most contexts, white privilege doesn't care about your actual ancestry. A 2nd generation black immigrant from Africa to NYC will face some of the same discrimination as a descendant from slaves. And with your white skin you will receive some of the same privileges as a wealthy descendant of the Mayflower.
It doesn't really hurt to recognize this, and it doesn't have to "erase" the pain that your ancestors went through. It's simply recognizing that there are inherent subliminal biases in our systems and society.