Apparently they break down in the body into hormones similar to estrogen, and is accused of causing hormonal imbalance in men and women.

Here's a good example of the anti-soy party line: http://www.menshealth.com/nutrition/soys-negative-effects

If it is really that bad is hard to say. There are plenty articles online that ask the question "will soy give you man boobs" and surprisingly you'll find Betteridge's law of headlines broken, as there are articles that answer this question in both affirmative and negative.