No man is island. So even if you are just harming yourself by drinking or taking drugs you are also ruining your family and indirectly society by making that behavior more socially acceptable. So there shouldn't be absolute freedom in anything.
I effectively never had one, so does that make it acceptable?
> and indirectly society
I know plenty of drug takers that function fine in society[0]. I anticipate your response might be "they would function even better if they didn't", but perhaps forcibly removing choice from adults is also damaging.
[0] and some don't, to be sure.
It's not even that there's no room for compromise, there's no space even to discuss it, to try and understand their rationale. It's like being a kid again in a classroom with the teacher shouting at you that you can't do that and no they won't explain why.
But even though this loose standard can be stretched to justify literally any prohibition, in practice the only activities that get targeted in the name of community enhancement are the ones that make the group in power feel threatened or uncomfortable.