The key not physically being there is unnerving. There's no tactile feedback, _unlike every other key on the keybord_. As a touch typist, I found this extremely distracting during the week that I tried out the MBP 2016. Not every attempt to hit Esc worked, so I found myself constantly and instinctively hitting it 2-3 times "just to be sure", and then also constantly glancing down to see whether in fact I was aiming right. That's not great UI.
Another problem with the touch bar is unintended touches. I constantly found myself accidentally resting a finger on the Esc key, which of course caused all sorts of weirdness in various apps.
Finally, as a touch typist the touch bar was simply distracting. I never look at the keyboard, but the touch bar really wants you to look down, which is a source of distraction, especially when it changes contents in the periphery of your vision.
If the touch bar brought lots of utility, I might have learned to deal with it, but it is really is of limited value. Emojis? There's cmd-ctrl-space for that (the touch bar doesn't have search so you'll be scrolling a lot to hunt down the right emoji). Safari tab previews? They're idiotically tiny. And so on.