I can see your point, but want to add why I recently started to add this meta-tag on most of my responsive sites:
1. images start to look blurry
2. input fields change the scale of the page and seem to confuse some users (at least in some test-cases).
On the other hand I try to improve accessability by making sites screen-reader-friendly and using high contrasts. I have also seen a lot of people using "bigger fonts" options in their browsers or on their phones, which solves this problem from my point of view.