I was trying to highlight that theres your value to your employer, which is kind of divorced from your pay. And the value of the job to you, which is your pay.
Could your pay go down in the example? Theoretically yes, in practice we see the opposite. Economic development couldn't happen if it did. Although I suppose short term dislocations could lead to lower pay in the short term, before increasing long term?
Don't forget 'progress' is people increasing productivity. In a world where everyone has to work full time digging potatoes, that's all anyone has. If it only takes 1 member of the group to dig potatoes, that frees the other members to discover science, religion, chips. We are in a situation now where a fraction of the population work in food production, but we all have jobs and more wealth than potato growing us could possibly imagine.
That added productivity doesn't disappear, in our earlier example the owner of the means of production has captured all the value that I didn't get from it. He has to do something with it. Even if he puts it in the bank, that's possible seed money for a venture that ends up employing my 9 ex workmates.
Also don't forget that the one potato digger won't want to dig potatoes all day whilst everyone else sits around. If there's no reason to dig more he'll dig enough for himself and take the rest of the day off. Now you could entice him to dig for you in exchange for beans, because you know he doesn't like picking beans, and as you don't like digging potatoes. And of course I could offer to trade apples...