The safety equation for men and women differs vastly. The pill is acceptable at its current safety level because the alternative is pregnancy which can be lethal. Plus the various uses in cycle control, important not just for fertility but also hormone management.
For men there is no downside to not taking the contraceptive, so it must boast a much higher safety profile to be considered. As it stands, the current attempts were either much too dangerous (higher rate of adverse responses than the pill) or too permanent, nothing to do with the fragility of men - perceived or otherwise.