I moved interstate for 3 months to take part in an accelerator, and had to drag my son and pregnant wife with me. That was hard (mostly on them!)
When I was pitching for investment in the US (we're Aussies) and missed my sons 2nd birthday, it nearly broke me in two.
When I went to the US to participate in 500 Startups, 6 weeks after my wife had given birth to our daughter. That was incredibly hard...for everybody.
12 months ago when I didn't take a paycheck for a month, knowing I was failing my family, that was the hardest.
Right now I'm in Sydney, away from my wife and kids, I will miss my Son's karate grading... and I feel like a terrible father for doing so.
But here's the thing. All these things are hard, and I do them because they have to be done. Not because "it might be fun" or "just to see what happens", but because these things will lead us to success.
The bar for what constitutes a worthwhile activity is raised so much higher when you have a family to support. You learn to be more critical on evaluating opportunities, and you only jump when you need to jump. It doesn't cripple you, it forces you to focus on things that move the dial.