This must be a recent change? It's been driving me nuts lately. I have to resort to adding a lot of negated search terms to compensate but it's still sub optimal.
But bejesus, this drives me nuts! If I know the double quotes function even exists, then Google should know I actually want to use it as intended - it shouldn't decide "yeah, but maybe you'd like these irrelevant results too!"
Regardless, I basically just always search with verbatim on. Google is mostly useless otherwise.