If device switching is more of an annoyance/issue than it is a pleasing feature, you can configure it. And it's configurable per-device.
When AirPods are connected, go into the bluetooth preferences, select "Options" next to the connected AirPods. Change the value for "Connect to This Mac"