In fact he said pretty much the opposite to the interviewer asking about magnets in this well known video: https://www.youtube.com/watch?v=MO0r930Sn_8
"Once I asked him to explain to me, so that I can understand it, why spin-1/2 particles obey Fermi-Dirac statistics. Gauging his audience perfectly, he said, "I'll prepare a freshman lecture on it." But a few days later he came to me and said: "You know, I couldn't do it. I couldn't reduce it to the freshman level. That means we really don't understand it." David L. Goodstein, "Richard P. Feynman, Teacher," Physics Today, volume 42, number 2, February 1989, p. 70-75, at p. 75
[1] http://en.wikiquote.org/wiki/Talk:Richard_Feynman#Unsourced
[2] http://en.wikiquote.org/wiki/Talk:Richard_Feynman#Teaching_q...
I have frequently heard the quote without that section, that section seems flawed in that a layman would need to have all the underlying concepts explained to them also, indicating that application of the quote would not scale, as more underlying concepts were necessary to include.