> - giving up the quality of your nice neighbourhood because a giant complex was built next door
> - giving up much of the capital appreciation in your house because it appreciated
> Both would be be highly unpopular for the majority of existing home owners. As a result everyone complains but politicians don't actually do anything and nothing changes. There is no such thing as additional affordable housing without any cost. There is a price to be paid. Most are not willing to pay it.
This is a pretty limited view, frankly. Your first point is spot on -- housing density needs to increase. But this does not mean worsening quality of life.
For decades density increases barely touched quality of life; a couple houses would be replaced with a tasteful 3-story apartment building, not "a giant complex was built next door" -- neighborhoods changed relatively slowly. Like Rome, Manhattan wasn't built in a day.
These days, in many places, there are exactly two practical options for construction: detached single-family homes, and massive complexes. Why? At least in California, because any kind of development not by-right will face years, and potentially a decade or more, of citizen action against it in every venue imaginable. If you're going to incur that kind of delay/anger anyway, you might as well build a giant thing and make your pile; there's no room for smaller buildings because they're not profitable enough to pay for the years of fighting the neighborhood groups.
Look at San Francisco, for example. Even if the population were to double, all you'd need is to double the housing units. This does not require massive complexes, it would be enough to convert single-family homes to duplexes, build ADUs, small apartment buildings where townhomes are, etc. -- except that those activities are more or less impractical.
So, we get today's dichotomy: single family homes, allowed by zoning, and massive complexes, conditionally allowed (requiring approval) because that's the only new housing that is worth pushing through the extreme regulatory environment.
Allow just a doubling of density in any SFH zone, and watch how quickly you end up with reasonable duplexes, ADUs, etc., that increase housing stock without a giant complex next door.
The only thing you have to give up is the fetishization of the single family home inside the urban core. Your neighborhood will stay beautiful and nice. And your existing home will continue to appreciate, because it will still be more desirable than the duplexes being built, which are in turn more affordable.