Don't you think there's some significant non genetic component to whatever made him so smart? Otherwise we'd be hearing about his relatives.
You'd expect for any of his kids to be outstanding, but not earth-shatteringly so. And looks like that's about right if you look up his daughter [1].
[0] Check out his "known for" section in Wikipedia! https://en.wikipedia.org/wiki/John_von_Neumann
[1] https://en.wikipedia.org/wiki/Marina_von_Neumann_Whitman
People greatly underestimate just how effective children can learn. They're basically capable of absorbing as much information as you can pump into them. It's a delicate balance making sure they learn enough basic human operating stuff in addition to everything else, but the results can be remarkable.
I remember hanging out at camp with a kid who loved board games. Especially probability based ones, such liar's dice and Dominion, but also more general hidden information games like Stratego. Kid was undeniably a prodigy, and he kicked my ass every time. I had just graduated university with studying CS and Physics, this kid was looking forward to starting kindergarten in the fall.