I wasn't careful enough with my language. Yes, someone can work harder and see diminishing returns anyway. My dad installed carpet for the majority of his adult life. I would characterize that as more difficult work than most, but he never attained very much wealth.

Still, in a (very particular) vacuum, I would argue that statement holds water, even if it doesn't in reality.