But I think you need to have a general affection for design in general to really get anything out of the book.
In my experience you don't :-) At least, I've seen it switch many people out of the design == "making things pretty" mindset into something more productive. It's always on my reading list for people who don't get design, or people who understand that they need to get design. Seems be effective with both groups.
I think it has something to do with it not being written as a "design" book per se. Remember the original 1988 title was "The Psychology of Everyday Things". It's up there with "Peopleware" as a book I keep buying for clients :-)