One of the most significant issues I see is there's often no way back into math if a student gets off track early on. My students typically got confused in 2nd-5th grade, and then math never made sense again. When given the space to work independently, with support, at their own pace, they get to clear up old confusions and realize how much they're capable of when they're not under pressure to keep up with everyone else.
One of the things I enjoy most is taking students who haven't understood math since 3rd grade, and bringing them to a place where they can start to use algebra to solve problems they care about. From then on, they are set up to make good use of math the rest of their lives.
I have a 3 1/2 year old son now, and it's fascinating to watch his development after having been a teacher for so long. He's in that wonderful stage now where "22" is just as big as "a million", and when he wants to describe a large number of things he just strings random numbers together: "Dad, there's 23 45 8 17 100 stars in the sky tonight!" Just the other night he asked, "What do you get when you add 3 and 3?" It's fascinating to watch that understanding develop, and to just keep answering his questions honestly, with the mindset of giving him a strong foundation for more advanced concepts later on.