It's likely because the microphone/input channel is being activated - forcing the low-quality audio codec.
As soon as the headset is told to turn on the mic, quality drops significantly.
I figured it was by design, and using a better codec/more bandwidth whenever it could afford to, but dropping down to half that in order to run the mic channel side by side.