>> Is it really just "I worked hard for it why should I give it away"
No it's more "I worked hard for it why should it be taken from me by force or fiat". I've found most folks who are well off (not living hand to mouth) are willing, and do give to charity. It's when some other group of people start demanding this behavior through governmental force when attitudes change.