The problem with some projects (e.g. GNOME) is that they consider all feedback to be equally invalid, and the self-appointed "designers" are always right. This is the result of the Apple-inspired worship of almighty "design," and is contrary to the very idea of personal computing empowering people.
IMO Gnome used to be significantly better than it is and everytime they break something they break it in the name of userfriendlyness.
However the real feedback comes from what is not said out loud. Users might dismiss the problems they have as their incompetence, but only you can be the judge of that. If 3 people in a row make the same mistakes, I can guarantee you it is not their incompetence you are seeing... ;) That is why observing the users is so important, much more than just listening to them.