In the four years since quitting my day job to freelance and build my app, I've gotten engaged, married, gone on a honeymoon, got a dog, gone to Europe for a week, and had our first child. I also regularly go on vacation and go snowboarding. I've been full-time on Sifter for about half of that time. So I was part-time for about two years, and that was hard work balancing the two, but it wasn't that bad. I just split time between freelancing and building the company. It was stressful at times, but well worth it.
In my case these days, as a solo founder, it's a bit involved. I work every day, but for the most part, I still only work 40 hours per week. I work from home and see my wife and daughter all of the time and find time to go to the gym every single day. I take breaks and spend time with them whenever I want, and I just work whenever it fits into the family schedule. The only catch is that it's very hard to stop thinking about work in my downtime, but I think that has as much to do with me and being passionate about work as it does being a business owner.
It hasn't been a cakewalk by any means, but it's entirely possible. I've got a slide deck that covers some of the personal side of things if you're interested. (http://bootstrapping.sifterapp.com)