But, let's flip it around. If your phone goes off and you ignore it then that sends a message that I'm more important than whatever else that could be going on in your life. That can even make our connection stronger.
I've personally used this to my advantage when talking to someone that I want to impress (e.g. a potential investor). If my phone is going off like crazy I simply apologize and tell the other person that I need to mute my phone because it's a distraction from our conversation and that his feedback is more important than whatever else is going on.