This is true, particularly, if you work full-time, and your job is not connected to the subject of your passion. You will have few hours in the evening, but for a hard subject you may not have enough energy.
You can try remote work, it worked for me. Currently, my client allows 2-3 days a week of working from home, and also now I’m looking for my next place for a full remote position. I save 1.5 hours on commute, 20 mins on lunch. So I have additional time and energy to focus on maths, building things, and running.
There is an additional barrier to move into a different or more advanced field: most companies want to hire people already experienced in particular areas. It will be very hard to convince a company that you can learn fast and contribute and also that they can already benefit from your current skills and technical maturity. It looks risky for them, they may have relevant candidates, or do need such expertise right now.
Our industry love talking about fixed vs. growth mindset, how crucial is the latter. At the same time, the hiring at many companies is a vivid example of fixed mindset towards candidates: your current skills matter infinitely more than you potential. The thing is it can also be hard to convince them that you actually can acquire required knowledge fast: you know, everyone says to be a “quick learner” in their CV.
I think your situation is typical and for some people the only solution was to start their own business eventually, and outplay the system.
So, focus on improving yourself in the evening and keep trying to find your way in!