That's not generally how it seems to work. For junior bankers under 30 (analysts and associates) you're making good money, but not such great money that you're putting away hundreds of thousands of dollars a year after your expenses and taxes. I can't recall ever hearing of someone saying, "I've worked 5 years, I've saved $500K, but instead of making $500K next year, I'm going to retire."
Also, the lifestyle (and partner, and kids) that you end up with while making banker money means that $1M is not remotely "quitting money", because it won't last long enough. I'd guess most folks would talk about $25M or $50M min for that in the industry, and you're not saving that anywhere below MD. Bankers have banker wives, banker mortgages, and their kids have banker tuitions. (One of my bosses had an $8M mortgage on a $12M home. That's ~$30K in mortgage payments a month... no way he could quit) I'm not saying anyone should feel bad for them, just the whole idea of "save up $1M and quit" doesn't make any sense in reality.
Finally, there's a lot of pride/ego/image folks have in their job and their job title, for better or worse. Even if it ruins their life. I didn't love being a banker, but I can't deny it's not fun telling folks you're an investment banker.
(Also, it's worth pointing out that people don't go work 100 hour weeks for years on end just so they can quit with $1M. They're doing it to have $10M or $100M.)