There are a few exceptions to this rule however. For example saponins. One of the most common plant poisons in the natural world that might kill your dog. If you ever read an ethnobotany piece and it mentions that a certain plant is used to kill fish in a pond, it's likely a saponin-rich plant (which also means it's likely to be good to make soap out of). But since these compounds are so common humans have evolved specific enzymes to break this down and they leave us mostly unaffected (indeed some have even been turned into important drugs).

It's kinda like lignin. When wood first evolved, there was nothing that could break it down. And for thousands of years the world was "littered" with wood from dead trees until fungi finally learned how to break it down. I'd say plastics might be the next lignin as we see more and more organisms already learning to break it down, but there are too many different types of plastics that I doubt all of them could be broken down

I think the point I'm trying to make is that the "toxicity" of a chemical or compound is often in proportion to how much time the rest of the ecosystem has had to evolve a way to deal with it