When I quit, it took guts, because a job in finance does give you a false sense of comfort and security. But looking back, I was infinitely happier from day one, and am better off than my peers who stayed in their front office roles (this, of course, is not guaranteed and ymmv). I also feel my potential to grow is much more than my past colleagues, who still seem to be trapped in their job's safety net.
Now I'm a passionate engineer and building good products makes me happy, so that's what I do. I have, through my company, launched a couple of private SaaS products, and hope to launch another, more public product soon. I also do a bit of property development in the UK. Whatever time is left over, I use to play tennis or spend time with family. It's great!