You'd be pretty annoyed if the front door to your house, when you opened it, also opened up your document safe, emptied your wallet onto the floor and invited visitors to leave bugging devices to spy on you later.
Also, the defence of "just don't agree to pair your phone with an unknown USB device" can actually be tricky. On a flight, I plugged my phone into the USB port on the seatback to charge it. The phone repeatedly kept asking if I wanted to pair it with something (who knows what it was? the entertainment system, maybe?). If I had accidentally hit the wrong button only once (on a prompt that randomly appeared), my phone could have been owned, and there's no easy way to un-pair.