See my other comment:
I recall AOC lobbying for stopping an apartment complex in a formerly industrial area. And a articles on how certain areas of Manhattan don't allow towers. In fact, a quick look at the zoning map [0] has, eg, large areas zoned as "R8B contextual districts are designed to preserve the character and scale of taller rowhouse neighborhoods".
SF is the worst offender, but NYC is still pretty bad.
It can be doing so much better. I get a tad excited reading about 1,000ft supertall's being built, only to sigh when I see it will have a whopping 80 units. I've seen even more ridiculous stuff like a 10 story building with 4 units. There's been a ton of construction in NYC over the last 5 years alone but it seems to be mostly luxury and medium/low density. There are plenty of older buildings from the 20th century that are much denser but it seems that no one wants to build these anymore. (Or they can't, I'm not sure which).
[1]https://www.nytimes.com/interactive/2016/05/19/upshot/forty-...
Additionally, the other boroughs aren't super dense. For example, much of the land next to LIRR is pretty low density, and even SFH [0]. Some people in Manhattan would be okay living there, so that does lead to higher rents in Manhattan.
0 - https://www.planetizen.com/news/2021/06/113861-new-york-time...
I say 6 stories because you can reasonably have a 6-story walkup. Anything higher and you would need to add an elevator which could make things harder.