The fear is that it happens at a subconscious level. Most people have subtle (latent) biases against disadvantaged groups.

A perfectly awesome to work at company might be doing this out of ignorance, not because they would be a bad place for OP to work.