When I see that word, "privilege," especially tied to attacks on white/male/hetero/whatever, my response isn't "oh jeez, not that again".....it's "yeah, that is a problem, and I really wish the other white male hetero folks in the room could see it and not just roll their eyes".
It sounds like you want to be an ally, and that's great. I think the hardest part is probably recognizing that when people talk about privilege, it's really not personal, it really isn't about you, even though you're a beneficiary of it every day. Knowing about systemic injustice and doing nothing/remaining silent, though, that's pretty callous, and knowing about systemic injustice and perpetuating it, well, then it would be about you.