But with maths I think it's a bit more subtle. Just the tiny things (even just 'talk' math) you do in early education can make a massive return later. There is no 'mean' in maths education even in ages 4-10 (my kids). We live in the Netherlands, that scores quite well in the table. And still I think math education is at least unambitious for the higher percentiles.
With just a few minutes here and there I've managed to get some pretty abstract concepts in their thinking (say: zero and the empty set, that's something small kids understand intuitively / or the first algorithms with russian multiplication or divide and conquer). There's no need for pushing, and still I think I've already help lay a good foundation of math in my 10 year old. No more than 30 hours invested in math his life.