When I want to talk to my phone or my watch I don’t say “Hey Siri” anymore, I just hold down the lock button (aka use it the old way). That activates Siri on the device without triggering any other devices. Since I generally have my watch or phone at hand, this has solved the issue for me. Still I wish the detection for “which device am I talking to” was better.
Also not all siri+device combos can do the same things. Our HomePod is not customized so anyone can use it for basic things like playing music but then it can't do personalized tasks like set reminders. Some apps only exist on some devices and a Siri request involving those will fail on other devices.
I've turned off "hey Siri" for most devices and only use it for the HomePods who really need it to be useful. For my phone or watch, I hold down the button and the whisper to the device hoping the HomePod down the hall doesn't hear and start playing something in the living room I only wanted to play in my office.
If Face ID gets a glance lock — I.e. looking at your iPhone directly — while speaking a command it “locks” onto that device (try that out by saying “hey siri” while using an iPhone). You can also just hold the side key instead of Hey Siri-ing it to not hop onto another mic.