I guess you don't drive to work if you somehow equate driving (something necessary to live) with taking mushrooms (unnecessary and risky pleasure)?
Who decides what is a necessity and what is not? I would say that the harm rate could be a good indicator what a society should tolerate and anything that is less harmful what is tolerated should be legal.