User-scaling of the viewport is thankfully not the only way of enlarging text inside of a web page. It's possible to design a site that allows the user to change the font-size of relevant UI controls without rendering the layout wider than the device screen (causing the user to have to pan around to read it).

If you're not providing a better experience than a fully working desktop site, then you shouldn't attempt responsive design at all and let mobile users use a non-targeted but working site. I only shut of user-scaling on sites where the targeted mobile experience is better than a scaled fully-working desktop layout.