The brain is fully developed by age 25 and starts to lose mass from then until death (see: http://www.ncbi.nlm.nih.gov/pubmed/19283594). Brains of 90 year olds can be just 60% the size of a young adult. Neurons do not repair well and they (or rather, stem cells) aren't programmed to divide after the skull runs out of room.

I would not worry, though. The brain is very flexible. If you keep your mathematics sharp, you will lose less useful things like old memories instead of skills. Many professionals work in difficult jobs (say surgery) well into their 70s by keeping their skills in use.