Even just to go to your mailbox? I routinely walk in the door from work, change into some more comfortable clothes (which may or may not have pockets), and then grab keys to go get my mail. I'd say I don't have my phone most of the time I do that.
But if my mailbox requires me to have my phone, I'd bring it along. I don't see the problem.
Both myself and everyone I know have their phones on them 24/7, and next to their bed when they sleep.
How come you don't forget the keys then? This phone thing is such a non-issue. You could even memorize the code when you get the mail and won't need the phone at all after that.
Like it takes two seconds in the app or you can block all notifications from that app by long pressing the notification.
Really not that hard to get rid of the spam.