Besides brushing up on my math, starting from simple arithmetic, In my spare time I also study Japanese. And one of the things that has helped me the most in my fluency and understanding has been the memorization of vocabulary and of grammar patterns and their usage.
Of course, I read materials at my own level and listen to material at my level and above and practice writing. However, I noticed the biggest boost in my comprehension after I memorize a large amount of words or really internalize grammar patterns. And I do this mainly through flashcards.
I have to spend much less mental energy to catch on to what is being expressed allowing me the ability to potentially comprehend more.
And analogously to my language studies, I would like to approach math in a similar fashion.