Look into the history. Caffeine wasn't under the radar, it didn't make the cut when drugs started do be banned in the early 1900s. It didn't make the cut because nobody abused it or formed a serious addiction to it, because you can't. Take too much and you feel like you're dying.
Coca-cola = coca leaves + kola nuts = cocaine + caffeine
Caffeine was everywhere too in the era of drug concoctions. Today's soda's were the health tonics of the 1800s. 7-up had lithium in it, the drug used to treat some bipolar cases.
The drugs that survived public use were the ones that were paradoxically safer and more dangerous at the same time, depending on how you measure it.