Wigner himself said the only genius amongst The Martians was von Neumann so he at least thought von Neumann was special compared to the others. For your example a mathematician who would fit better would be Norbert Wiener, who was taught from a young age by his father to be a prodigy.