You don't have to have a phone on you all the time. But if you do, it should be considered bugged because you have no way to audit it.

Also, bugging my residence isn't going to result in very interesting data by design. I could have plenty to hide, but I'm not reading my evil plans for world domination aloud.

If you do need to talk about something private, nobody is preventing you from disconnecting all the microphones.

Know what you're trying to prevent, then take actionable steps. Privacy is hard, but not dead.