That's a valid point, I think, but I am not sure the line is as clear as you make it out to be. What is your definition of 'directly' here? I can think of ways in which e.g. hunting endangered species, owning weapons or eating meat affect others. Note that, of those: I revile one, I am undecided on one and I personally do another, but in all three cases I can think of ways they affect me when others do each of those things. Sure, they affect me in a way that is perhaps different than being the target of murder or slavery would, I'll definitely agree with you on that. But it is a leap from there to saying that those are matters society should not be involved in.

Consider nuclear proliferation: in theory it does not affect people 'directly', until it does (note that any logic about increased potential risk applies to gun ownership too, at a different scale). Now consider pollution: affects people directly, but in a diffuse degree. Vaccinations? (I was born in a country where a specific set of those are mandatory, I approve of that [1]). There is a limit to how much a man is an island.

I am a firm believer in having a right to privacy; a strong, fundamental, constitutional right. I would consider forms of deflection or civil disobedience in a society that aligned itself too strongly against that right. But that doesn't mean is not ultimately a matter of societal values of societal construction.

[1] Obviously they are mandatory to those with a healthy immune system who can be vaccinated, and a number of other caveats, but still, the state makes it its business whether or not you vaccinate your kids, and we are better off for it...