There's a financial interest. Cheaper homes means your home is worth less. There's a tipping point, however. There's a point where you can no longer afford your home or the neighborhood.

There's also a question of quality of life. A prison is one, extreme example. But an apartment building aimed at college students or young professionals might be another. High density housing might create more traffic, noise, parking problems, property issues, mess, and so on. As someone who recently moved out of a "low rent" neighborhood, I had my property thrown up on and vandalized multiple times, and been woken up in the middle of the night more times than I can count. I was not sad to move to a neighborhood not far away, but significantly quieter. I realize these problems are not related to the property per se, but certainly people chose a neighborhood for a variety of reasons that are not aesthetic.

This isn't to dismiss the need to improving access to housing. I don't want to be NIMBY, but we seem to have this immovable dichotomy between liberal "expand access" policies and conservative "if you don't like the neighborhood, move" policies and no real way to reconcile them.

EDIT: I have a short anecdote to try and illustrate my point. In my former neighborhood I was lucky to have a small, one car garage on an alley. Parking was at a premium and most residents in the converted rental housing (houses that were turned into multi-unit rentals) did not have parking. At least a handful of times per year, I would find a car blocking my garage. Perhaps I didn't leave for work until later in the day or something, but whoever needed that spot thought they were OK there for a while. This is not a trivial problem for me. I needed that car to get to work and now I either have to track down the owner or wait for a tow truck (and incur the owner's wrath which usually involved yelling at me from my yard or smashing my mailbox) and be late for work or wherever I was going. The problem is that this wasn't a serious issue to most people reading an article on HN (or even the local paper). It's not an issue that you make policy based on, but it is an issue for everyone who lived in the neighborhood, including the people who were forced to block my garage for lack of other parking. I can't call the cops and tell them I think my irate neighbor smashed my mailbox or call my city and tell them to do something about parking. I'm stuck with that problem until the slow wheels of planning fix it, and by then I've been late to work a dozen or more times.