There may not be societal costs in the consuming country, but there are certainly negative externalities to the people/countries that produce the coffee beans
But if we want to use that metric, OK. I would venture a guess that the production of coffee yields a heck of a lot less suffering than the production of cocaine, heroin or meth. Any far fewer people consume the latter.
In any case, legal production can be regulated in ways that illegal crops cannot. It's hard to draw a legitimate comparison.