This is usually the sentiment of the top comment on HN whenever teaching children comes up. However, in this case, I don't really see the down side. They still get to do all the time things you mentioned, but they can also read. I find it hard to imagine the risk of alienation to the point of being labelled as misfits and somehow stunting their social development is high enough that it's worth worrying about.

We could just as easily speculate that these children can help their friends understand the world and become closer with peers as a result.

We aren't talking about locking them up for years to focus on reading, it's just a bit of spaced repetition.