"Phone-as-camera" - people have always traveled with cameras since it became practical. They took crappy pictures of fireworks, or whatever, on film. Perhaps a bit more mindfully because of the limited amount of exposures on a roll of film, but it's still a major way to capture a memory. A shutterbug would spend just as much time shooting film and not being "in the moment" as someone with a phone. It's just that now phones work as pretty decent cameras along with all their other features.
"Phone-as-nav" - this is a great advance over paper maps in that it tells you where you are on the map.
"Phone-as-comms" - also a great advance in that you can get in touch with travel companions or make contact with locations.
"Phone-as-distraction" - aha. THIS is where I think the issue comes in; if you are too busy tweeting or reading email or playing Clash of Knights while you're visiting the Pyramids or the Forum Ruins, THAT'S a problem. That's where you need to put down the phone and be a part of what you've traveled to ostensibly see.