I think it's more that it's implicitly derogatory to other men. People get annoyed when you speak for them, sort of deal.