Of course we all have a stake in what drug are illegal, what gambling can be done, what prostitution is acceptable - because I don't want to live in a society of drug addicted, gambling addicted, hooker addicted people. Do you think we're just atomized individuals who shouldn't care if your neighbor is caught in self-destructive behaviors they can hardly escape due to addiction? Do you think that doesn't affect you? Do you not care about your community and how healthy your community is?
This libertarian idea that "it doesn't matter if others are engaged in the most unhealthy behaviors because its not you!" needs to die, because the end result is a shittier society we're ALL forced to live it.