The great irony is that high density housing dramatically increases the value of the real estate, which is the actual investment. A single family home is in fact a wealth sink - you're constantly maintaining it and even with great effort it will probably be an undesirable building to live in within a few decades. While someone may pay to live in the building you currently occupy, at some point someone is going to buy the land to do something else with it. Like imagine if you owned an acre of downtown Manhattan with only a ranch style house on it - the skyscrapers around you are not going to lower your property value.
There is concern that lower rent places will allow lower income individuals to reside in these communities, and that will lead to undesirable conditions like crime, despite the fact that if rent were lower than a person of a given income level would have more cash in pocket and thus be less inclined to things like crime, and nearby affordable housing means local businesses could find people to work for lower wages, lowering the cost of living for everybody.
If people invested in making their communities desirable places to live, genuinely increasing the value of their real estate, rather than relying on scarcity to increase the value of their house, everyone would be better off.