Drugs are a public health problem not a criminal one. People are taking things like heroin primarily due to other factors in their life, which the state should address. For drugs like magic mushrooms lack of availability is increasingly looking like actively harming the availability/development of a positive treatment for many conditions.
[0]: https://www.theguardian.com/politics/2001/jun/14/drugsandalc...
(tldr the negative impacts of heroin all arise from its criminalisation, while it is extremely addictive it's also extremely hard to overdose on when taken on its own)