Also working with some of the smartest people in the industry can be intimidating: if you're used to being the smartest guy in the room, it's very unlikely you will be anymore at a top notch firm. This is a form of stress in itself.
It's not stressful in the sense of shitty clueless bosses, long unpaid hours (people work long hours, but that work is often highly compensated for through pnl improvements).
Everyone I know feels extremely lucky that by just putting our heads together, we can mint money out of thin air. I don't think anyone I work with would rather be doing anything else. Trying to make money out of nothing certainly humbles you, but when something works, its the best feeling in the world.
Keep in mind that things might be different at a larger firm: lower compensation and upside, more hierarchy, etc.