For what it's worth, I don't think it's too late. Your biggest challenge will be to take it seriously, put in the time, and to be resilient in the face of the difficulties.

But I have a friend who is in his late 40s who is coming up to speed in Number Theory, and doing well. But he is serious, and is putting in the time.

With age comes maturity, that can help, but also a more set way of thinking, which doesn't. Also, it depends on what kind of mathematics you want to learn ...

... and why.