> to me, is fucking terrifying.

The non-intuitive thing about basic reading and math skills is this:

If you delay the learning, you learn it much more quickly.

So the (ordinary, with no developmental issues) kid who doesn't read until 10, can exceed his or her peers in just a couple years.

The very unfortunate corollary, however, is this:

The kid with developmental issues might go undiagnosed for a longer time than necessary, with long-lasting repercussions and follow-on effects. It's even possible to reinforce a mild issue and turn it into a larger one.

This then becomes a classical optimization problem. Optimize for the common case for broad benefit, or optimize for the rare case with significant negative effects. There is no right answer for everyone.