As someone who's studied the topic quite heavily, here's my issue with no-holds-barred legalization (I consider "legal" to mean "can manufacture and distribute" in this regard, like tobacco). Drugs that couple very tightly to the VTA (brain's behavior-reward circuit) actually can lead to lasting changes after few or single exposures. This is amplified by rapid absorption routes - injection, insufflation and inhalation. The main players here are heroin (DEA sched 1), meth (2), cocaine (2), GHB/Xyrem (3), some benzos (4), tobacco (legal), and analogs therein. It's hard to have a casual habit with these drugs, as they manipulate the brain's behavioral calculus itself.
Note that the DEA schedules are all over the board. The current schedule system has barely any correlation with harm (see work by David Nutt).
But we are also in a weird place where access to two of the most destructive drugs in real measurable effects - alcohol and tobacco - are legal.
At bare minimum we need to overhaul the schedule system, legalize drugs with low/moderate addiction potential (weed, psychedelics, ketamine, phenethylamines, the Shulginoids), and legalize everything in personal possession quantities, and for scientific/medical research.
High-addiction drugs should probably be medically gated in some way.