I'm currently working part-time on retainer, which allows me to work less than full-time and still make enough money to live (definitely less than a full-time job, but enough to live comfortably). This leaves me time to build my own thing while not living off my savings.
I don't know what I'm doing, but I might suggest something like this. Stay at the job for now. Use the financial security to buy a house and get a good mortgage. Then start working on taking part-time work on top of your full-time job. This will suck at first (full time plus extra work), but once you feel comfortable about being able to get part-time work, you can feel comfortable leaving the full-time job.