Privilege is a word loaded with connotations about being born with a silver spoon in your mouth.
White privilege isn't about being upper class because you're white. It's about the advantages a white person has over non-white people of the same station.
A white person in a poor neighborhood is on the whole more likely to be able to pull "themselves" out of poverty, because, the systems and power structures in place have white people in power, or have systems that are fair on their face, but still lead to higher white hire rates or social mobility.
Nobody is disputing that some white people have it bad, they're simply pointing out that all else about your social status being equal, being white provides tremendous advantage over being non-white.