It may not be the most harmful drug, but it's certainly not harmless or non-addictive.
LSD can cause mental illness to emerge in some people. I also remember a study that gave LSD to 10 people during a religious sermon. 9 of them went on the be preachers or priests later in life. That doesn't necessarily mean it's bad. But anything with that powerful an effect on the mind is concerning.
Amphetamines certainly seem pretty cool. Especially if you have ADHD. But it definitely has side effects and potential for addiction. It's long term effects are not well studied, and a few studies I did see were concerning. E.g. brain damage in monkeys, and a group of kids taking it did worse than the control group after 3 years.
I think people should be educated on the risks and facts before they decide to take a drug. If they still want to do it, fine. But at least they know the risks. And it's not enough just to put a warning sticker on the back of it that no one reads. Even widespread education that a drug is dangerous isn't enough. Everyone knows cigarettes cause cancer, but why do new people take up smoking?