Occasionally my wife and I talk about my quitting work, and us moving to Thailand (her native country). Her vision of my ideal life there is that I wouldn't have to work, and could sit around doing my pet projects without any time or money constraints.
That always strikes me as dangerous. Those projects I want to work on didn't jump into my head unbidden. The ideas arose from years of coming to grips with real-world problems that don't have entirely satisfactory solutions. If I untether myself from the work that led to the ideas I want to pursue, then where will the next ideas come from? And how will I know whether they are worth pursuing?
To me, having side projects that seem worth doing, especially from the point of being useful to others (Sivers' way of distinguishing what he means by "not just relaxing"), requires their being grounded in real-world projects. And one measure of the worth of real-world projects, like it or not, is that people are willing to pay to get them done.