You've thrown out two contradictory statements. If you agree with zoning in certain circumstances then you can't hold that property is a right for use by owner as they see fit(without the threat of being kidnapped or stolen from). Just because the current zoning system in most locales is overly bureaucratic, or even corrupt, doesn't mean the idea of zoning should be done away, or that property is a right the government can't interfere with.
Criticism is one thing, but framing this as an issue about the government taking away our rights is not productive.