I suppose you didn't buy the entire town, so you can't preclude your neighbors from wanting their property to appreciate, their lives to become more social, and their environment to change.
The neighbors collectively agreed to the previous zoning. Now some outside body is forcing that to change.
Once upon a time, neighbors also collectively agreed to zone by race and ethnicity, until Buchanan v. Warley in 1917 put an end to that practice and created the minimum-lot-size SFZ proxy system we have today. You can argue persuasively for respecting SFZ, but you probably can't do it from a moral high horse (which is just to say: you can't rest your argument on a moral right).
I’m not sure the implication that those of us who don’t enjoy sharing walls are immoral racists is particularly justified.
It's a historical fact. I'm not saying you're a racist. Those zoning ordinances though definitely are.