Einstein himself even said (although it's laughably inaccurate) that he wasn't smarter than anyone else, he just stuck with problems longer than other people.
That line of thinking seems very...well...stupid.
Einstein had a difficult time accepting parts of quantum theory, but I don't think you can say "the world passed him by". He was generating fruitful scientific ideas well into his 60s, if not until his death 1955 at age 76 (and some would argue, as would I, that his contributions to ethics and political thought much later in his life were as important as much of his science).
For example, in response to the Copenhagen interpretation of quantum mechanics, Einstein, Boris Pololsky, and Nathan Rosen devised troubling thought experiments into the late 1930s (the "EPR paradox"), which, while surprisingly not coincident with experiment, led to such famous results as Bell's Theorem...called by some among the most profound ideas in science.
http://en.wikipedia.org/wiki/Bells_theorem
I don't believe many of Einstein's biographers accused him of hubris...or of simple ignorance.