Are you struggling to keep your head above water? If so, why not just focus on work? If not, then why don't you just spend less time working and more time doing what you want?
I believe people have limited capacity, trying to stretch yourself will result in burn out. If you can afford it, allocate some of the capacity you are dedicating to work to the non-work things you are interested in. If you can't, then just focus on work until you can