On the extreme end, you can say a design definitely doesn't work if a large percentage of users can't accomplish their goals or get frustrated in trying, but you can easily run into situations where half your users love a feature and will be frustrated without it, and half hate it and will have fits if you don't change it. Of course, these are the kinds of issues UI designers are paid to resolve, but there will rarely-to-never be total agreement between a designer, a client, and the client's customers on which aspects of a design work and which don't. Not so in development.