More recently, with much better French, I've had people impatiently switching to English the moment I was struggled to find the right word.

There certainly is a shift, though, when you get outside the tourist areas.

E.g. in Nice, you can pinpoint it almost by street. Cross under the railway in the city centre and move into Liberation, for example, and you will immediately find some staff that are unable or unwilling to speak English, and many who at least won't switch unless you ask. But it's also not surprising given how abrupt the number of tourists fall off...