Far more men like women with good-looking fit bodies. It's a statistically insignificant minority that likes "fat women", so "men" overall aren't policing women's bodies to the point of influencing women to not go to the gym. It's largely women who do it to each other.