Why workers producing goods that exclusively go to benefit the upper class be considered a good thing?

If there were no idle rich to make expensive products for, (say they were all beamed off the planet to work in an alien slave mining facility) the end product that was produced from any job would defacto have to help those in the same or lower class of the worker, benefiting everyone.

In other words, what's better for the world, yet another rich man's yacht, or cheaper, better tasting loaf of bread that a poor person can buy and eat?