You mention sterility. As a thought experiment, what if the science was perfected that you could change gender perfectly (ala The Culture novels). Would you still feel the same way about it?
It'd be prohibitively expensive for most, but why not?
Also, you mentioned environmental factors, is that thinking based on a paper or similar?