What would you do if you didn't have to work? Spend more time with your wife and kids? Isn't it a bit odd to say "if only I spent less time with my wife and kids, then I could have a chance to spend more time with them in the future!"?
Spend X hours working per week to get Y hours of family time
vs.
Spend X+n hours working per week (or just X if you can find ways to be more productive) to some much larger value of Y hours of family time over some estimated larger timeframe.
It's all a gamble and about risk, but if you think you might be able to retire early in say, 10 years, and get to spend a much larger amount of time with your family (and be presumably less stressed and in a better position to support them financially) that might be worth a bit of extra hustle or focus on productivity than spending 20 years working for some similar result.
I'm being really hand-wavey with numbers, but when people think of early retirement for these reasons, that's often the thinking behind it.
There's a fair question to be raised on the quality of that time spent, particularly when you look at your health when you are younger, and the impact of spending time with children when they are young, but there's a lot of factors that people consider when weighing these questions, and I don't think it is entirely fair to judge someone because they may weigh those questions differently than you did.
For me the question isn't what would I do if I didn't have to work. The question is: What if I could have absolute control over what I do and wasn't beholden to anyone else because of rent and food and such?
That would be neat. I happen to find getting there (working more) more enjoyable than watching Netflix and other time wastey things so it works out. I enjoy my work so I want to do more of it. And I want to do less of the work I don't enjoy.
So the equation for me is more: Spend a little less time with wife/kids today to improve my future enjoyment of the work part of life.
It's a tough balance, because I highly value wife/kids time -- so when I give up some of that (valuable and well-spent) time for a potential future good, I want to make sure I'm using that time optimally, so that sacrifice is as "worth it" as possible.
There's definitely a balance here; I could make much faster progress by working every night and weekend, but that would be too much of a sacrifice of the good present for the potential better future.
Anyway, I very much agree with/appreciate the point of your questions :).