In the counterfactual, where these hard-working poor people are lazy instead, they aren't suddenly rich or even the same as they are now. They're almost certainly worse off.
I was raised with a strong work ethic. Do I believe that working hard is the only reason I am where I am now? No, of course not. But if I didn't work hard my life would be worse in every way imaginable! Regardless of how lucky or unlucky your life is, having a solid work ethic will always always always make you happier. Work is not inherently 'bad' in some way, and neither is having to work to live and to learn and to have a good life.