That's a strange way to look at the situation, when you consider the larger picture of a man's struggle in life.
Someone who's working at Walmart does not make a lot of money, they most likely cannot afford to give their children the best education, they cannot afford a reliable car, they're more at risk of failing to make the rent payment due to some unforeseen accident.
Someone who's in the financial position to be able to comfortably invest their money, or is otherwise a millionaire, even if they lose a good few hundred thousand dollars, it will not affect them much. At least, not as much to pose the question of if they'll be sleeping on their own comfy bed or in a homeless shelter.
When you say that people "deserve" minimum wage because that's all of the value they're providing to society, this is what you're talking about. You're talking about putting them in a very shakey position, you're talking about making them borderline-homeless.