It's not like I don't have the time, it's the fact that I'm way too tired after work to do any substantial amount of learning. I think I could manage something that's much easier, say learning some languages or learning how to knit. But learning math, physics, etc just requires more than I have to give after work.
That said, I managed to get a firm understanding of orbital mechanics up to a level where I could probably ace all the tests for freshman-level courses at an aerospace university and I did all of it on Saturday afternoons (but it took me 5 years to do so).