I think the middle ground is that you have to be careful about what zoning and planning codes do and don't allow. In my view, a property owner like you should have pretty broad discretion about how a place gets developed. A lot of very nice-looking town centers developed in the late 1800s/early 1900s (North America) without zoning. It got done because it worked, more expensive/taller/nicer buildings near the middle of the city (on more valuable land) and less-expensive, "lower" uses toward the periphery. It's simple market incentives. The trouble comes when the tax code, zoning, or other policies discourage the natural, higher-intensity development from creating more housing on higher-value land. You're just looking at the incentives and being a good businessperson. People (blue collar) want a place to live. You're giving it to them at a price they can afford. This should be celebrated, not vilified.
I also own (market rate) low-income housing. In the neighborhood where I grew up, there are a bunch of lots with older houses sitting vacant. I own an apartment there and would love to convert some of the existing (empty) houses into duplexes/four-plexes, but can't out of a misguided sense of it being a "single-family neighborhood". This makes no sense. I'm turning something people don't want, into something they do. The zoning completely ignores market forces out of a misguided attempt for neighbors to force their view onto other property owners.
It's well-intentioned but, as other posters have said, carries a huge cost we need to talk about more openly.