But when your neighbors bought the land they bought it priced with the knowledge that their neighbors (you) could not do that, so to them, you can suck it.
(I'm for relaxing zoning laws but if you ever wanted to convince anyone this kind of argument is like the last approach you should use.)