The reason this is done is to get rid of extra latency when processing user input. I guess detecting zoom gestures needs some time to distinguish them from normal clicks?
* Some number of people are sensitive to latency
* Some people are unable to read my website
You're gambling that there are more people who would walk away due to latency, than there are who would walk away due to being unable to interact with your site at all.
Personally I suspect that to be a poor tradeoff.