Is having to pull your phone out of your pocket that bad? Do you even need to look at it unless it makes a loud noise at which point you can close to defer until later? Constant distraction is poisonous. On your arm, it's going to be worse.
Going back to my airport example, I generally put my phone in my computer bag when traveling to avoid accidentally losing it. This is a case where not having to pull out the phone right away is extremely useful.
Other activities that benefit from not taking out the phone: grocery shopping, shoveling snow, working out, cycling, riding public transportation, cooking, and I could keep going.
Constant distraction is another conversation altogether, but I generally don't find this to be an issue by configuring my notifications to only show things I probably need to know about right away.