Except most babies seem to be fine, so maybe the fear is overblown?

I think it should be kept in check, as it seems all sorts of nasty stuff can be mixed into plastic. But many plastics seem to be fine.

There is also poisonous stuff in plants, including wood.