There is a wide gap between not wanting to be responsible for helping the poor and actively wanting the poor to fail. You're confusing the two.
There are genuinely many people who wholeheartedly believe that the poor deserve to be poor, and that helping them is bad. Some of them aren't even that well off themselves, but have bought into an ideology that's detrimental to them.
If you haven't encountered these people, then count yourself lucky, but don't try to deny their existence or assume your own experiences are universal.
You can't deny the politics of retribution exists, because politicians only give oblique references to it; voters certainly believe it, hence one voter who complained about Covid shutdowns thusly: "He's not hurting the people he needs to be"