Can't speak for OP, but I started learning algebra at 5. It happened that I really liked playing with calculators at age 2-3, and that helped give me an extremely solid mental arithmetic foundation.

Essentially, the special thing about me probably wasn't some super-human neural circuitry for understanding math. It was the fact that I was wired, somehow, to enjoy playing with a calculator. I had more arithmetic practice by age 4 than most kids have by age 11.