My desire to do this though, causes the problem that I feel like I can't stop working and constantly increasing my savings.

The idea that if I take time off for a while to work on something at a loss, the damage will be too great due to the long time effects of compounding interest etc and the desire to build a prosperous life for my family.

So I think I have the opposite problem. I have a hard time stepping off the gas pedal to pursue things that would probably be more beneficial and enjoyable.