The concern about soy in infant food is well placed, but your broad statement is hyperbolic.
I don't know where you are, but I'm in the US, where 2/3 of adults are overweight, and half of them are obese. Whatever effect soy has on our endocrine systems, it is dwarfed by the effects caused by body fat itself.
As for the effect of soy on adults, for most people there is no problem unless they have excessive soy consumption, and those problems are fixed simply by eating less soy. Soy also has known health benefits, and an honest discussion would weigh the two.
From 2018: https://www.ncbi.nlm.nih.gov/labs/pmc/articles/PMC5646220/
"Neuroendocrine disruption by soya isoflavones in mature neuroendocrine systems is by and large reversible with dietary modification and thus, with the exception of some hypersensitive groups such as hypothyroid and oncology patients, soya likely poses no long term health risk and may even confer modest benefits."