I've quit my Software Engineering job twice now to spend years of my life doing whatever I want. First time I spent two years driving from Alaska to Argentina [1] and second time I just spent three years driving right around Africa [2]. Both times I had no job, no boss, no timeline and I did whatever I wanted every single day. Hiking, camping, taking photos, cooking food, learning a new language, writing when I wanted to, etc. etc. I had ultimate freedom, living off my savings.
Without a doubt, the "freedom" life is much harder for me than the "desk job" life for exactly the reasons you listed above. After about six months for me I started to really, really think about the meaning of my life, what direction I want my life to take and a ton of other things that simply don't enter my mind when my life is on autopilot (i.e. going to work every day).
Going to work everyday is easy, because it means you don't have to decide the direction of your own life if you don't want to.
[1] http://theroadchoseme.com/expedition-overview
[2] http://theroadchoseme.com/africa-expedition-overview. & http://instagram.com/theroadchoseme