Common misconception: Most estrogens (particularly phytoestrogens, but also plastics) bind to receptors so weakly, they work more like antagonists in the presence of stronger estrogens (like red meat).

It'd be like trying to do anabolic steroids while your natural testosterone is very high.