I agree. Even with my nice sound isolating earbuds (Shure), it doesn't block out all the external sounds unless I set the volume higher than I would like.
While playing music on headphones can help block out conversations, it is still a distraction. It's just not as annoying as hearing other people talk.