Most "side effects" of psychedelics occur in the short-term, while under the influence of the substance. I will focus on LSD, as it's the substance I am most familiar with. The side effects are mostly what people call the "body load". Most notably there is vasoconstriction, increased body temperature, increased blood pressure and heart rate, and of course, the possibility of a bad trip or triggering latent schizophrenia/psychosis or bipolar disorder in those with a predisposition.

Edit: There are also interactions with certain medications that can result in things like seizures, and very rarely people report "flashbacks" or HPPD.

Overall, in my extensive experimentation both with traditional psychiatric medications and with LSD... I think the side effects of the substance are overstated. I have far worse reactions to psychiatric medication. (Edit: your mileage may vary of course, different people will react differently to substance, illicit or otherwise - I don't mean to downplay the negative experiences that DO occur with psychedelic usage)

This is something that can be googled, but personally Hallucinogen Persisting Perception Disorder (HPPD) is the one keeping me from ever trying them.

Take shrooms for example

Now, I’ve never seen any issues with microdoses and would have no problem suggesting such to anybody. In fact I’m a big fan of broadly studying and implementing them as medicine. But what I’ve unfortunately noticed as you go up into high doses (determining actual psilocybin is difficult per mass, but this is a decent reference: https://healingmaps.com/shrooms-dosage-information-guide/) you eventually start seeing cases of… well I’m not sure exactly what you call it other than a psychotic break or injury.

Basically you know how some people seem to have various mental chains broken and are now able to live and enjoy life better because of their trip? The opposite can also happen.

One victim was a long time user that went from gradually from a medium to high dose. Then on the next move up in dose they got fucked. Basically described it as essentially having their connection to everything good in the world severed, feeling like a dead ghost trapped in a corpse, incapable of happiness or love or connection with reality. They could function and appear normal, but were in a constant sort of experiential agony. This wasn’t just a bad trip, this was their daily life for several months. After seeing doctors and trying various things to fix it, they committed suicide.

It doesn’t matter how long one has used lower doses, or how gradually one eases into it, a very small portion of the population seems to get severely injured. For the sake of example, even if such an occurrence is 1 in 10,000 or 100,000 that’d still be unacceptably high to broadly recommend a dose with that level of risk factor for the sake of having a trip or healing. But that’s also more than low enough for someone to truthfully believe and say “dude that’s not a thing no one I know has ever had an issue”

Edit: turns out it has a name: https://en.m.wikipedia.org/wiki/Hallucinogen_persisting_perc...

There are also many cases that occur where someone under the influence can become temporarily dangerous to themselves and others. Basically the idea of say attacking themselves or others or some other dangerous activity enters their head, but there’s no longer the voice that immediately say “nah”, and instead they just go with it. No malicious intent, it’s just impulse with no filter, like water flowing downhill. This is much more common in comparison to full on breaks, but usually isn’t a huge deal as the person is subdued by friends and no one really broadcasts such an event, but occasionally injury and even deaths do happen.

Once you get past low doses, you’re playing chemical roulette. There’s an extremely high chance of winning, but losing does happen.

The “psychonaut community” being what it is, is loathe to admit these types of problems exist, and will generally be dismissive or hostile and accusatory if it’s brought up.

There’s also the issue of their being no good way to get hard data on this either, and determine if there are other risk factors involved that would allow certain people to know they have a hidden life destroying landmine and to stay the hell away from experimenting.

There may possibly be more subtle medically negative effects that could show up with chronic use, but I can’t really speak to that.