When I quit, it took guts, because a job in finance does give you a false sense of comfort and security. But looking back, I was infinitely happier from day one, and am better off than my peers who stayed in their front office roles (this, of course, is not guaranteed and ymmv). I also feel my potential to grow is much more than my past colleagues, who still seem to be trapped in their job's safety net.
Now I'm a passionate engineer and building good products makes me happy, so that's what I do. I have, through my company, launched a couple of private SaaS products, and hope to launch another, more public product soon. I also do a bit of property development in the UK. Whatever time is left over, I use to play tennis or spend time with family. It's great!
As an aside, the software engineer complaints about technical interviews always makes me laugh. Finance ones were even worse. Mostly useless and much more intense and time consuming. Engineering is tough because it's hard to point back at prior performance in a tangible way. Anyone can sit in a large corporation and write some for loops. It's much easier to hide in the background in software. Trading was easier to vet: "someone gave me $30m and I didn't lose any of it".
So it is with money and finance.
It's valuable for middle/upper management to have someone 'technical' in meetings. Often as a sounding board/is this kind of thing possible/how should we do X/can you explain how Y works/explain it to me like I'm 5. You're providing what feels like 'general knowledge', but is actually knowledge/perspective gained over a number of years working on/in technology.
And depending on the industry, you might also be wheeled between projects and end up having the same conversations again and again just with different people.
From a personal perspective it feels like you're not learning anything (and you probably aren't), and it's a waste of your time. Rather than explaining to people the difference between message queues and databases you'd rather be building systems and solving 'hard' problems (either problems that are hard, or not involving 'soft skills' :-)).