Our ability to determine the toxicity of a compound is always directly proportional to our ability to concentrate it. Many synthetic compounds trigger concern, distaste, and hysteria simply because we are able to measure their effects precisely, while many "natural" compounds slide on by with error bars a thousand (or million, or billion) times wider. This is irrational.

It's reasonable to avoid risk by sticking with (e.g.) tried and true food sources, but I'd argue that the revealed preference is for a much larger amount of risk tolerance. The most ardent naturalist will gladly accept a new berry that none of their ancestors ever encountered simply on the basis that other people are eating it without ill effects. They do not demand generational studies on millions of people, and yet the same concerns you bring up for synthetic compounds absolutely apply. Nature is responsible for the overwhelming majority of the worst toxins we know of, but unknown natural compounds are accepted with a relatively modest amount of evidence for their safety. I think the double standard we apply to synthetics is absurd.

See also: radio waves. We can concentrate them to the point where they literally vaporize living tissue. This means we can make (and have made) absurdly good measurements of their toxicity and carcinogenic tendencies, yet my (techie!) friends still joke that their cell phones are giving them cancer.