It doesn't bother me if an applicant pool is made a bit bigger for marginalized people than myself, because I do not struggle to fall into any applicant pool whatsoever and have not for most of my adult life. Be it for employment, education, or housing.
Where conversations can get annoying is when I feel demonized because inequity exists independent of my individual actions. But I can live with someone bitching on Twitter because I'm not afraid of the police or telling potential employers I have a kid on the way.
What I don't like living with is the inability of others like myself in my own caste who would dismiss reality on reductionist notions "hiring by race or gender is inherently racist" without critical thought. Ignoring lack of diversity tacitly supports inequity and is itself racist or bigoted, by virtue of doing less than nothing to correct it.