After that it's like everyone tells you--choose the most important things to do with your time. My advice would be to make really sure the goals you are choosing match with what you're internally motivated to do.
It may be that your spouse and children (and maybe even your day job) are more important to you than side projects. If that's the case, and those things eat 100% of your time, be happy that you're doing what's most important, and stop wasting energy fretting about the endless list of things you would do if you had no time constraints.