There are plenty of people who dislike responsive designs for good reasons. It is not the fault of responsive designs itself but the fault of many mobile browsers like Mobile Safari and Chrome for iOS. Here is what I mean:

Responsive websites usually don't let you zoom in by panning. Everything is fixed. If you want the text to be larger you are out of luck on the browsers mentioned above. On iOS you cannot make the font size bigger in Mobile Safari. You can only turn on zooming in the accessibility settings which has its drawbacks as well. I like to zoom in to the relevant parts of a website myself - manually.

If most mobile browsers were capable of letting the user pick a different font size then I would be all for it.