Also, the fact that you describe it as a restriction probably reveals a bit about your age =). (I don't meant that in a bad way).
I think a lot of it comes down to life stage.
If you're 24, with no family or commitments, life is easy and you can stay at work late, or code till 3am if you want to - the worst is your room-mate (or parents) might complain about the noise.
Hypothetically, if you're older, and have a wife and 3 kids...well, those things are going to take priority (and they should). Of course, they can provide an immense amount of joy and fulfilment - at least for me, far more than my work can (although I do really enjoy my work as well).
I have worked in IB, algo-trading, and now in tech - and none of them compared to the pure ecstatic joy I get from coming home, and making dinner for the kids, or reading to them, or helping them brush their teeth.
I do still code in my spare time - but honestly, with a family, there may not be that much free time (assuming you still want to keep other hobbies and pursuits as well).