It didn't feel distracting the way receiving a phone call while being remote felt. It's clunky enough that people didn't come up unless they had to and I still could keep up with the verbal information that was passed along with the office gossip.
Some people get really bothered about this idea and probably if you are working on independent tasks you don't need to be so connected. For tight teamwork I found it to be the least bad option.