If you'd like a more specific example of what I am talking about, take for example Bisphenol A (BPA), one of the monomers used in making polycarbonate - which most people have heard of, and has now been eliminated from consumer products like water bottles (edited per correction from reply below). It is chemically similar enough to steroid hormones, that it can bind to steroid hormone receptors (in particular estrogen receptors), making it a potential endocrine disruptor. HDPE does not contain complex functional groups that are likely to mimic natural hormones and bind to receptors or enzyme active sites in a living system.
In general, molecules that contain a lot of aromatic rings, or reactive functional groups are more likely to exhibit drug like effects or toxicity. Such properties are systematically enriched for in chemical libraries used for drug discovery, but if trying to avoid unintended biological effects, it makes sense to exhibit more caution with such molecules.