It seems like magic. As everyone doing magic tricks knows there is a lot of effort going into a good one. The "innate abilities" or "genius" is the "magic" we all like to experience.
In the case of "von Neumann" he isn't isolated.
He is often lumped together with other Hungarian Jews fleeing Nazism and refered to as A marslakók ("The Martians"): Erdős, Kármán, Pólya, Szilárd, Teller, Wigner ...
Hungarians are a curious bunch, aren't they?
Arguably the strongest female chess player of all time (in the same class as the strongest men) is also Hungarian: Judit Polgár. How do they do it? There is no magic, no genius, her Hungarian father wanted to prove exactly this to the (western) world (in the east the notion isn't prevalent). And he simply conducted the experiment[0] consequently his youngest daughter became a so-called chess genius. Or as a Korean or Chinese might say she is doing the hard work.
We all like to tell ourselves stories and it is fine, it is a great and very effective way to convey certain things but sometimes we lose sight of what is really going on because of our infatuation with the narrative devices. Instead of learning and trying to explain things, we sometimes just want to indulge in the "magic moments". But as Feynman put it nicely: you can do both one doesn't exclude the other.
[0]https://en.m.wikipedia.org/wiki/L%C3%A1szl%C3%B3_Polg%C3%A1r