Y U no responsive design?
Responsive websites usually don't let you zoom in by panning. Everything is fixed. If you want the text to be larger you are out of luck on the browsers mentioned above. On iOS you cannot make the font size bigger in Mobile Safari. You can only turn on zooming in the accessibility settings which has its drawbacks as well. I like to zoom in to the relevant parts of a website myself - manually.
If most mobile browsers were capable of letting the user pick a different font size then I would be all for it.
It sounds like what you dislike is poorly done RWD, which only addressed the aesthetic of the small viewport and not the unique navigation & content challenges (unfortunately common).