At Harvard it really is easy to stop worrying, if you have an open mind.
As a mathematician, I know excellent academics who narrowly missed getting tenured at MIT and Princeton. They were highly in demand, and are now happily working at other top-20 universities.
(ed: I am in math, and described what I observed. I see elsewhere in the thread that mdwelsh observed the opposite.)