There would be fewer holdouts if anti-vaccine activism would not be a lucrative market for fraudsters. But ultimately this is natural selection. Civilization kept stupid people alive, now there is a counterweight against stupidity.
Sounds sociopathic, but beyond incentives selection processes shape the world the most, and we are grossly out of shape.