You might be taking the quote too literally. I think what von Neumann was getting at is that the abstract world of mathematics is essentially artificial and alien to our natural sensibilities and that if you expect to 'understand' something to its very core you may be misguided. The deeper you go the more counter intuitive things can get.

It reassures me because it feels at times that the people around me who are so 'great' at mathematics are born naturals and that I may as well give up. For me is a great 'leveller' to hear that even the greatest minds struggle with these things, albeit to differing degrees.