Does it bother you if a male candidate gets a position that maybe there's a better qualified female candidate who wasn't considered because "maybe she wants to start a family" or some bullshit? Do you let that (potential) bias affect your judgment of whether the guy "deserved" it?
Biases exist. Meritocracy is an illusion. Deal with it.