I agree with that, but their actions do impinge on my preferences. My preferences disappear the moment theirs are exercised.
Here's an example. I absolutely do not believe that this is actually a fair comparison. I do believe it follows the same logic as your example.
Let me explain what I mean. Let's say that I lived in an apartment, and developed a habit of listening to extremely loud music with bass starting at 10pm, and ending at 5am. My neighbors have a "preference" to be able to not hear that loud music at all, and certainly not when they're trying to sleep. They may not _like_ the loud music, but that's as far as the harm goes. To enforce _their_ preference, they would have to restrict my behavior.
Now all reasonable people would agree that playing deafening bass in the middle of the night is inappropriate, and that it would be perfectly fine to restrict my behavior. In other words, we restrict others' behavior pretty frequently. And, this restriction is based on "common sense" judgements. When the preferences of somebody falls outside of the norm under this regime, they may not have much recourse.
Now to be clear, this is far from a perfect comparison: there are laws against noise pollution, and property lines are not really the same issue as "being obnoxious on purpose." In other words, I agree and admit that these two scenarios are not interchangeable in effect. (I do believe that they follow the same logic, however.)
It's also not the case that in the "duplex" scenario that the greatest ill is not simply that I must "see" the duplex. Really, population density in an area has a number of effects:
- More people means more noise pollution. (and in some cities, kids will play loud bass in their cars no matter what time of night it is)
- More crime. (if not per-capita, then in absolute terms)
- More trash, debris, and pollution in my general vicinity.
- Longer travel times.
- More crowds.
- Etc.
And again, I'm not suggesting that I should have a legal right to prevent density from occurring. However it seems that when density occurs I have no recourse but move. I don't have any way to preserve something which is singularly important to me.