To be fair, privilege isn't only based on skin color. Only you can decide if your colleague was out of line in that instance, of course.
Consider that most people here on HN have some level of privilege simply by being in the tech field, which is highly paid and respected right now.