Sometimes all it takes is another couple years of slogging away at a job you hate to get you to a critical mass where investment income matches your yearly expenses. At that point all risk of quiting is basically gone, and you can pursue whatever you want.
Also keep in mind that seven years is a pretty common number in business for a good product to become successful. You can have 100% growth year over year for 5 years and still only have 32 customers. So it may make sense to hold off until you have slightly more runway than 5-6 years savings.
But doing the math, and putting dates on calendars can do wonders for how you feel about going into work.
Also, you sound like you're in a really good spot no matter what you decide. Good luck!