I think that perhaps it alone should be restricted in the same way that access to plutonium is restricted, with the aside that individuals who would like enough for their own personal use should of course be granted (free) access.
You see the problem with that particular problem child is that enough LSD to dose every man woman and child on earth could be produced in a single lab and held within a coffee cup which makes it uniquely dangerous as a weapon of mass oppression if it were to fall into the wrong hands. Also the fact that you can't easily test a substance for LSD increases the liklihood of individuals being exposed who did not wish to be exposed.
Therefor the risks that it will be used as a kind of weapon against unsuspecting populations is too great relative to other drugs. I'd also like to point out that LSD doesn't seem to universally help people in the way that responsible amounts of MDMA (and arguably cannabis) does.
I don't think there is a justification for the restriction of other psychedelics that don't have these risks of weaponization such as various mushrooms, cactai, certain vines, etc.