For males not to have nipples, they'd need to be actively destroyed, which poses a risk for females to also not have nipples, which is much worse than males having harmless, inactive nipples.
The amount of testosterone in women is not zero, likewise the amount of estrogen in men is not zero as well, and breast tissue does serve some purpose in regulating hormoe production, even in men.
Mammalian fetuses all start out the same and sexual dimorphism happens several weeks into development. The same structure that eventually develops into a uterus can instead develop into a penis/prostate. Testicles and ovaries are the same tissue early in development, just like the glans and clitoris.
Biology doesn't generally suppress one entire set of organs in favor of another. They're built from the same precursor tissue and only diverge after sex hormones are activated. Biology and evolution modify existing structures, it does not typically erase one structure and replace it with another.
In addition, intersex humans exist. There are documented incidences of males born with uteri, external genitals can form halfway between male and female. Biology can get very messy sometimes. Sex is not a hard binary switch, it's a sliding scale just like most biological features. Only most individuals are at one end or the other, there's a lot of room between.