You're not nuts - this has been a huge issue for a long time. It's what makes taking calls outside on a street so frustrating (or even turning on the faucet to wash a mug while on a call).
It's weird because this wasn't an issue with the initial versions of AirPods firmware. But somewhere along the line, they tuned the firmware to introduce the issue you're describing.