Stop feeling guilty for setting aside time to learn skills during regular working hours. It took me a while to adjust, but it's utterly logical and correct that technical people should explore technologies as part of their work. When I became a manager I encouraged all my teams to set aside time like this. I basically said: if the topic you want to learn more about is very relevant to your current task, that's great, if not, that's fine too, if it turns out to be interesting, put on a lunchtime seminar for the group so we can all learn something.

I think as devs we all know that we will sometimes go yak-shaving just as an excuse to dig into something we're interested in (or at least not return from the rabbit-hole with any urgency), so I'd rather my teams do this consciously and think about what might be useful or just fun and interesting to other people on the team as they explore.

Family time should be family time, and if you're waiting for it to end so you can get to your hobby project you're not going to be as present as you should be. Also, it might work a little for the first kid, but when the second arrives forget about it.