Using Fussell's class breakdown [1], it is likely that you haven't worked for any people in the "upper class"; you have met the several varieties of the middle class. Fussell's definitions of the class is more about their tastes and behaviors than about how rich they are.

The upper classes would never get on their hands and knees and pound nails, because they feel that it is beneath them. The uppers are content with who they are, and don't give a damn what the lower classes think. But they are also uninterested in new ideas and would not be curious about your profession.

The rich people that you work for are likely to be upper middle class. They are likely to have earned their wealth by working hard, so they can relate to the hard work you do, even if their work was "white collar" instead of "blue collar".

The "middle class" that you describe seem to be Fussell's "middle" or "high proletarian". They are near enough to your class that they are bitter to you because they feel threatened by you, or because they think that their snobbery makes them better than you (it doesn't — usually this is the mark of a person with low self confidence). They might also not be there when you work, because are also at work.

[1] http://www.amazon.com/Class-Through-American-Status-System/d...