> while many of my colleagues are at the top of the world

The grass is always greener on the other side. I promise you the work at these "high-quality" places is no more glamorous than that being done at other places.

Maybe the position at the other firm, although smaller, might end up being a different, maybe even better experience?