I quit my investment banking job in 2017 and set up my own SaaS. Very tough for the first two years, making little to no money (but I had a good amount of savings). Four years since quitting, I'm better off (many times over), both financially and in terms of how much time I have to do the things I want to.

Not for the faint hearted, but I definitely recommend it.