This concept is more or less in line with the spirit of the Bill of Rights, but seems to be curiously absent from common American discourse.
Enforcing ANY law, even a seatbelt law, restricts rights and causes some amount of harm. I find it perplexing anyone thinks legalized recreational substance use would actually cause more harm than drug enforcement does, even aside from the metaphysical harm of having your freedom reduced.
It is as if the harm inherent in law enforcement doesn't count, as if there are kinds of acceptable and unacceptable violence. Which doesn't seem like a philosophically defensible ideology to me.