Just plug the adapter into the end of your headphone jack and leave it there. It's a female adapter, so it can just hang off of your existing 3.5mm jack. You make it sound far more annoying than it actually is.
It adds too many unnecessary steps.
Adapters suck for unpredictable situations but those aren't common enough and Apple knows this.
Using a pair of headphones with multiple devices? At least between phone and laptop? That's not a corner case, even though Apple seems to think so by skipping out on a Lightning port on the Macs after putting out lightning headphones.