That's pretty much where I currently am. "easy" job, the rest of my free time is spent on open source side projects. But it's just not a great feeling to be paid to make nothing useful, and to have so much of my life be effectively wasted time.
Einstein worked at the patent office while coming up with relativity. Aristotle charged big drachmas for tutoring Alexander. It may not be the dream, but there are undoubtedly many more examples of people who managed to roll the peanut of humanity a bit forward despite doing something else to pay the bills.