That's circular. They're treated differently because they're treated differently. But why do we put so much effort into maintaining it when it seems pointless?
Arguably, almost all cultures and religions have rules related to nudity, sex and in particular, females bodies and their sexual behavior.
Or are we just saying that women's breast are disconnected objects from the woman, and its those that we treat as objects to be consume? And in contrast, mens breasts are not disconnected objects from the man and thus are not to be consumed.