I'm almost 50 and have been doing my "own thing" full time since 2004. I'm not rich, but have been able to be home while my 3 daughters have grown up. I spent 18 months traveling around the country with my wife and kids (we homeschool). I've been lucky enough to work on the things I want to work on, for the most part when I want to work on them. I have had years where my income has been under $12k and years where I've made over $300k - it's been an absolute roller coaster ride. No doubt if I had stayed employed by 401k would certainly be bigger, but the "cost" to me has been 100% worth it.
One thing that really made the decision easier for me when I left my job to do my own thing was that I could have been in many different situations around the world where I would not have even HAD the opportunity to do my own thing - and by NOT doing it I felt it was disrespectful to all those who would take the chance if given the opportunity to be in my place.