I think the main issue that propels high-achievers into these careers is simply peer pressure. I was never happy in finance, my stomach churned as each summer internship approached, but I did it all because I couldn't imagine what people would think of me if I didn't get the next big internship or full-time offer. Did he get cut? Did he fail the interviews? Is he not smart? The pang of euphoria from getting the offer had to overcome the everyday loathing of the work and, of course, it never did.
Even if you don't believe most work in finance to be difficult, enjoyable, or even that lucrative you cling to the prestige attached to the name of your employer like in no other industry. When you ultimately work up the hutzpah to leave and see your friends moving on gaining new titles, different employers, etc. you still feel pangs of their relative prestige increasing despite all your internal protestations that you don't care anymore. It's insidious and comically absurd to try to explain to people (as I'm sure you've noticed). These vaunted colleges really do a number on many of their students.
Would be happy to hear more specifics of your story.