Some points are that it's going back up again and is largely caused by poor sleep health. Beyond that, I haven't looked this up but what is the actual relationship between lower sperm count and fewer children? I mean, you have maybe 2-3 children and millions of sperm, surely they are not linearly related.
Effects on mothers could matter a lot more, but also maybe they just don't want to do it.