So you can say that " All of us deserve to have fun once in a while, regardless of socioeconomic status." But for someone who is working too hard to take the time to have fun, this rings kinda hollow. And I can bet it stings a bit to have your taxes going towards paying for someone to do the things you don't have time to do yourself.
I went through a stint of working in cafe's recently, and one always had a group of people who would turn up after leaving the job centre (you have to go here to qualify for benefit money) in the early afternoon, and would just basically hang out. Frankly I had to fight the jealousy, I wish I could spend every afternoon hanging out with my friends. I had to remind myself how much it sucks to be unemployed, and how nice I have it. I can imagine this could seem unfair to someone in a more stressful position.