"White privilege" isn't an accusation, it's a cultural reality, along with all sorts of other types of privilege. The point of invoking the concept is not to make people feel bad about being white, but rather simply understand that society isn't actually meritocratic. Opportunities arise via combinations of what you've done, unearned privilege (who you are), and random circumstance.

If you're a white guy and you climb up for poverty, be proud of what you accomplished. Please just don't look back at everyone who hasn't made it and think that they all could have done the same.