Even midtown Manhattan has zoning restrictions, which is crazy to me. People like to talk about preserving neighborhood characters -- well the neighborhood character of midtown (say 30th-50th excluding east of 3rd) is tall buildings and streets as canyons. That should be a free for all except for preserving a few key buildings and their environs (e.g. grand central).
Similarly, the telos of the LES and EV are cheap houses built as densely as possible at the time they were built to house poor immigrants. You honor that history most by allowing dense housing not by preserving ugly block after ugly block in situ.
By all means preserve Striver's Row and some of the nice block of Greenwich Village but there's no good reason to keep large parts of Manhattan static. Let Manhattan be Manhattan, if you want static move to some quaint village in Westchester.