If the developer can get permission to build 3-5 story apartments on the lot

wouldn't that only work for a single unit? Like, yes, if inside a single-family-zoned block a unit would get a special permission to be the only one to build high density, sure that piece of land would be more valuable than before.

But if the whole block gets rezoned, then there's a race to the bottom.

Many single family home owners will strenuously object to having a developer build a 5 storey apartment building next door because they don't want additional noise, traffic, and people looking down into their private back yard. It directly impacts their quality of life.