Yup, this seems to happen whenever AirPods convinces itself that it needs to use its microphone, which seems to necessitate degrading the audio quality.
As soon as the headset is told to turn on the mic, quality drops significantly.
I figured it was by design, and using a better codec/more bandwidth whenever it could afford to, but dropping down to half that in order to run the mic channel side by side.