Put another way:
Bob lives on 200 acres and wants to construct an apartment building on that land. Nobody can really see it, he has agreed to provide parking and services, etc.
Sally owns a small residential house in a historic neighborhood. She wants to tear down her house and make a narrow four-story apartment building. She can't provide any room for parking and depends fully on city utilities.
Sally has more impact on her neighbors than Bob does. It's reasonable for Bob to merely have a question of general freedom. Sally on the other hand is clearly impacting her neighbors - and not every neighbor could do what Sally wants to do.
Some freedoms can impact the freedoms of other people, and that is when the consent of those other people is needed.