We need a social context for how and when to take them (beyond "let's partee!"). If you look at cultures with a long history of psychedelic practice, the use tends to be very ritualized, with a lot of specific context as to when they are taken, how often, and for what purposes. There is also substantial effort taken to integrate the experiences.

If you just take a bunch of stuff at a party and believe everything you experience is literally true, sure you are in a risky position. We need to push people towards communally-meaningful experiences with integration afterwards, and away from just blasting their minds into space willy-nilly.

We can't do this while they are illegal though. Keeping them illegal encourages the worst kinds of uses, with no oversight or integration or cultural meaning.