Do you not have family accounts set up? Each person in your home should have their own Apple ID and the HomePod for each person should be set up by the primary user. You can also set up the recognize by voice features and that should prevent the situation you're describing.