Well, if ugliness weren't enough, imagine if they instead did everything to make things unpleasant for people living nearby.

Since that would reduce demand in the area, there's a real risk that we'd infer that the neighbourhood is more inherently undesirable and drop the rates there, even though we shouldn't.

I think you're really basically just saying "but no one would do this". People would absolutely do this. People even do this today in inter-group conflict about who is to live in specific neighbourhoods.