There is a large gap in life expectancy between the richest and the poorest, not to mention the quality of that life. Poverty is not just a deprivation of possessions, it is a deprivation of housing, education, employment and health care. Increasingly it is a deprivation of clean air, water, healthy food and physical safety.
Poverty is a ruinous gravity well that chews you up and spits you out as a twisted shadow. It traps people in vicious cycles that can last for generations. You can escape it alive, but you can't escape it unscathed.
If nobody would work, billions of people would starve within weeks. The only relationship to other people that would prolong ones life would perhaps the ability to eat some of your peers.
Well that does sound like "bullshit" as you put it, but it bears no relation to what I said.