Feels pretty legit though. My freedom-from is impacted by other people's freedom-to: by curtailing their freedom, mine is expanded. Sure they won't like it - but I don't like it the other way round either.
I've watched it play out on my mother-in-law's street. What was once a quiet dead end street is now a noisy, heavily trafficked road because a large apartment building was put up at the end.
The number of freedom-to people have significantly decreased her quality of life blasting music as they walk or drive by at all hours, along with a litany of other complaints that range from anti-social to outright illegal behavior. Even setting aside the illegal stuff, she is significantly less happy living where she is now.
Consider how little freedom you would have if laws were enforced to the lowest common denominator of what people find acceptable.
For instance I lose almost nothing by not having the freedom to carry a weapon (UK) as I have no desire to do so, while gaining a lot from having the freedom to not risk my child being murdered at school.
It's an extreme example but applies to a lesser degree for other freedoms, and I've personally found I often benefit more from freedoms-from than freedoms-to.
I'd love it if no vehicle could exceed 30 mph in town as I gain almost no benefit from being able to do so, while taking on significant risk from others being able to.