They aren’t toxic or poisonous, but psilocybin does have a nauseating effect on many people. The cell walls of the mushrooms can be hard for some to digest as well, which is why many people make a tea or tinctures. You’d have to consume an absolutely insane dose of them — so much so that the current lethal dose isn’t known — for safety to be a concern at a toxicological level. Emotionally and mentally, different story.

In fact, they are typically considered the safest recreational drug, both in terms of its pharmacology but also in terms of reported ER incidents: https://www.theguardian.com/society/2017/may/23/study-halluc...