Fertilization is actually an accretive process. While only a single sperm will actually fertilize the egg, a single lone sperm is not capable of accomplishing the task. The ovum has a layer of protection that prevents access. Sperm cells, in turn, have some sort of enzyme payload that breaks down this barrier. It takes a lot of this enzyme before the barrier is weakened enough for a sperm to gain access. Not enough sperm, no one's getting through.

Additionally, the female reproductive tract is generally a hostile environment. This is by design, in order to weed out sperm cells with defects. Since this is a biological system, you're going to have some variation and some females are going to have a lower sperm survival rate than others. I have no idea what the range is, but I would be entirely unsurprised to learn the variation spreads over several orders of magnitude.