You seem to have really misunderstood the essay, then. He uses the word "idleness" to be provocative, but he's not actually saying that people find fulfillment in sitting and staring at the wall. He explicitly calls for efforts to be made to help everyone to learn enough about a wide enough variety of topics that they can choose their own interests to pursue in their leisure time.

From the essay:

> It will be said that, while a little leisure is pleasant, men would not know how to fill their days if they had only four hours of work out of the twenty-four. In so far as this is true in the modern world, it is a condemnation of our civilization; it would not have been true at any earlier period. There was formerly a capacity for light-heartedness and play which has been to some extent inhibited by the cult of efficiency.

> ...

> When I suggest that working hours should be reduced to four, I am not meaning to imply that all the remaining time should necessarily be spent in pure frivolity. I mean that four hours' work a day should entitle a man to the necessities and elementary comforts of life, and that the rest of his time should be his to use as he might see fit. It is an essential part of any such social system that education should be carried further than it usually is at present, and should aim, in part, at providing tastes which would enable a man to use leisure intelligently. I am not thinking mainly of the sort of things that would be considered 'highbrow'. Peasant dances have died out except in remote rural areas, but the impulses which caused them to be cultivated must still exist in human nature. The pleasures of urban populations have become mainly passive: seeing cinemas, watching football matches, listening to the radio, and so on. This results from the fact that their active energies are fully taken up with work; if they had more leisure, they would again enjoy pleasures in which they took an active part.