The problem with peddling guilt is that not everyone subscribes to it. I pay taxes, and I expect the needy and the poor and the sick and the hungry to be taken care of. If anyone should feel guilty, it should be the government leaders and the people that don't hold them accountable for the duties there are tasked with performing.
I think stealing is wrong, no matter the situation and desperation. However, I also believe that the supermarket (or whoever was stolen-from) has the right to compassion for this individual. And if you ask me, I believe people are generally good, and most would forgive the transgression without hesitation.