1. You breath on the thing, and it does something. It's extremely sensitive
2. It's really hard to fast fwd/rewind just a little bit. Its good at scrubbing through a whole movie, but try to go back 30 seconds with it, it's nearly impossible. Even worse, if you accidentally touch it, the movie goes fwd/back, and you try to scrub precisely to where you were - pretty frustrating.
3. You're not touching the thing you're interacting with, and unlike a touchpad, you don't see a cursor you're moving. It's supposed to be 'like' you're touching the screen itself.
4. Most people who aren't somewhat tech savvy just will give up (I set my dad up with our roku elsewhere for this reason). They expect physical buttons and to be able to 'click' through channels. It's not even apparent until you use it for a while part of the remote is a touchpad, as it's all black
It's like the remote was built for savvy iPhone users, not people that want to just watch TV.
Apple TV has other usablity problems, but that was a big one for me that would make me question getting another one.