Everytime I've seen something about plastics being a health risk, it's not the main polymer (eg. PET, LDPE, HDPE, etc.) that's pointed out, it's the plasticizers. Because plasticizers are not chemically bound to the polymer, it does make sense that they are more likely to have a biological effect than the polymers, as the polymer is generally a fairly stable molecule chain.
The polymers themselves can degrade, though. Especially with plastic items that have been exposed to the sun for long periods of time, the polymers can break down. I wonder if this degradation could also be a health risk, as many polymers are hydrocarbons, which do have known health risks.