I had the same problem but in reverse. I’d want to play a song on the HomePod but my iPad sitting on the coffee table would start playing it on its tinny little speaker. The solution was to set Siri on the devices to not respond to “Siri”. To use Siri on the handheld devices I press and hold the side button until I get the ugly gradient border. It’s pretty bad that Apple released this feature in such a uselessly broken state.