The "drugs" of alcohol and caffeine are more deeply integrated into public social life. Wine is mentioned in the Bible. Beer is sold at family outings like baseball games. The White House serves alcohol at official state dinners. President Obama had the famous "beer summit" https://www.google.com/search?q=Obama+beer+summit&tbm=isch

Also, the existence of alcohol is caused by common natural processes. E.g. leaving apples or pears out on the kitchen counter too long and it naturally ferments which creates alcohol. I remember leaving some old pears in the kitchen and they eventually smelled like alcohol. I ate the pears and they definitely tasted like alcohol. The point is, alcohol can come into existence without even doing anything.

In contrast, the White House isn't handing out crystal meth or heroin at state dinners and there aren't any natural decaying processes that turns random food into ecstasy. Marijuana is natural but it isn't culturally accepted by the public. (President Clinton's "Yes, but I didn't inhale.")

Alcohol/caffeine have a lot of acceptable functional uses that the other drug categories don't have which is why society continues to talk about them as a separate class.