They're treated differently because western culture treats women and their bodies as objects to be consumed and nothing more.
Arguably, almost all cultures and religions have rules related to nudity, sex and in particular, females bodies and their sexual behavior.
Or are we just saying that women's breast are disconnected objects from the woman, and its those that we treat as objects to be consume? And in contrast, mens breasts are not disconnected objects from the man and thus are not to be consumed.