As little as 20 years ago, spraying your crops with DDT wasn't illegal. Look how far we've come.
As little as 5 years ago, selling furniture treated with brominated flame retardants wasn't illegal in the EU. Look how far we've come.
We are wrong about chemicals all the time. The regulatory status of chemical X fifty years ago rarely means much.
But before we get rid of the current legislation about drugs, we would need an actual plan for dealing with irresponsible uses of drugs.
If people could take heroin or ice and (a) stop whenever they wanted and (b) still function as society expects then nobody would have an issue with them being legal. But drugs like those effectively rewire the brain and are incredibly destructive. A significant amount of domestic violence, child abuse, family breakdown incidents are directly attributable to drugs. You can't blame society for not wanting drugs like those to be readily accessible.
Regardless, the current legislative frameworks around the world (with few exceptions) are based on fear and a twisted sense of morality rather than actual harm.
I still got clean, because the risk of getting caught and the impact that would have on my future outweighed my addiction. I'm glad I did, and I'm lucky Australia treats it as a medical problem, not a criminal one, when you seek help.