That's a rather simple-minded response. You define the person as having privilege without knowing said person, and then dehumanize them by saying their situation is irrelevant to justify your offensive attitude.
There are circumstances that the purpose of removal of privileges is specifically for oppression.