I kinda have the opposite problem. I turn my phone to silent because I'm at a restaurant or cinema or whatever, then 3 or 4 weeks later discover I never remembered to turn it back.
And I'm 100% OK with that, I've gotten good enough at turning the ringer on when I'm expecting a call, and not worrying about "missing calls" from people who aren't in my VIP list or who don't leave voicemail.