The responsible use of drugs for your own use doesn't affect anyone except yourself. The chemical uses you brought up affects more than just yourself.
If people could take heroin or ice and (a) stop whenever they wanted and (b) still function as society expects then nobody would have an issue with them being legal. But drugs like those effectively rewire the brain and are incredibly destructive. A significant amount of domestic violence, child abuse, family breakdown incidents are directly attributable to drugs. You can't blame society for not wanting drugs like those to be readily accessible.
Regardless, the current legislative frameworks around the world (with few exceptions) are based on fear and a twisted sense of morality rather than actual harm.
I still got clean, because the risk of getting caught and the impact that would have on my future outweighed my addiction. I'm glad I did, and I'm lucky Australia treats it as a medical problem, not a criminal one, when you seek help.
But before we get rid of the current legislation about drugs, we would need an actual plan for dealing with irresponsible uses of drugs.