Some people can use these things responsibly, but probably fewer than the number who think they can and many cannot at all. That's why doctors are there to help guide responsible use. I agree it isn't ideal for the subset of people who can use them well, but there are also further issues: your drug use doesn't just affect you but the culture around you. I think the analogy to food is pretty bad, since no one is born dependent on amphetamines and it is a manufactured product only invented recently (to be fair, too much of our food is also weird synthesised products which are not historically natural, but I dont think you have a strict natural right to frozen microwave dinners either). The analogy with food may work a little better with mushrooms, but also the effect of these is partially due to your body thinking it has been poisoned, so they arent really food either.