You mean like drug abuse?
If you want to have ten sexual partners at once, do a ton of drugs, generally live a hedonistic lifestyle, you do you. I'll judge you and won't associate with you, but I'm not going to stand in your way.
The ability I have to leave my house and walk around is of a different quality to the ability I have to go to the supermarket and buy some food. Positive vs negative rights, something like that. I'm feeling lazy in my explanation but probably you know it already. :)
With drug abuse I feel it's quite simple. If you want to do meth in your home, I think you're a bit daft but I'm mostly happy for you.
If you end up in a tent city under a bridge abusing passers by and committing crimes to get your next hit, well, we should punish that hard.