Lets speculate this is entirely nurture. How do you get as smart as von Neumann? Zettelkasten?
I'm always super-suspicious when I hear someone's brilliance described in anecdotes. Some people are better at promoting themselves than others. So perhaps judge him only by his published work and not his reputation.