The trouble with working for yourself is, your boss can be an asshole. It's hard to take that vacation when there is billable work available. You can burn yourself out quite fast.

After having done this for a long time, I've learned to be zen about the intrinsic variance in income. But it is still hard psychologically. While rationally I know that I make more per year as a consultant and have more interesting work as a result etc, it's still painful to see the numbers in the spreadsheet go down when I'm not working for a few months, even if it was a deliberate decision that I made.