I recall running into women in CS in the 80s, far before the days of diversity hires. It was instant respect, as you knew that that person was not only legit, but had probably overcome extra obstacles to get where they were.
These days, anyone who could have been a diversity hire is guilty until proven innocent.
It's like a person brand new to the job - you don't know their skills yet. After a few months one's co-workers should see those skills in action enough that the suspicion disappears. And after a few years the worker will hopefully develop enough that they can demonstrate their competency to pretty much anyone.
And ultimately the person will be enjoying their career, which makes most other considerations irrelevant.
It's not as if outsiders to a job have the ability to accurately evaluate the person doing the job. A company can't be held responsible for someone else's Dunning-Kruger effect.
Do diversity hires get any special treatment after a no-hire vote?
That says a lot more about you than it does about diversity programmes.
If it makes you feel any better, as a rule I give diversity people higher ratings for similar performances.