You can argue that it's impractical or misogynist, but does it really baffle you? The double standard exists because (in American society) female nipples are sexualized in a way that that male nipples are not.
Arguably, almost all cultures and religions have rules related to nudity, sex and in particular, females bodies and their sexual behavior.
Or are we just saying that women's breast are disconnected objects from the woman, and its those that we treat as objects to be consume? And in contrast, mens breasts are not disconnected objects from the man and thus are not to be consumed.