You're right about idleness, it's like a person going rotten from lack of structure. Feeling bad, seeking cheap pleasures, going crazy.
But the alternative is leisure. Being able to spend hours playing music, writing or drawing, having conversations with friends, or watching the clouds. Building something for fun, or gardening. Making videos or youtubes, anything!
What causes someone to be idle while another person enjoys leisure? It's hard to say, probably a combination of their peers, their upbringing, their moral character.
But not working is definitely NOT a universal bad.
This is precisely why I didn't want to work in a factory when I was young. I lived in a town surrounded by the shells of people that used to have life in them, but after 20-40 years of boredom, repetition, and merely being a piece of the machine had severe effects.
Work may not be an evil in itself, but the culture of it really can be and most work out there simply isn't a rewarding experience outside of getting money. I'm not convinced that this sort of idleness is preferable outside of having money.
If the culture should change, it might be different. Encourage outside life, make policies so that folks can have family time and take care of them, discourage overtime: make sure folks have wages and free time enough that they can do something after taking care of things they need. This won't fix the horribleness of some jobs, but it will trap folks a little less.