Try working on the game 20 hours a week while keeping your job. If you truly love the concept and love working on it while still working full time, that'd be a good sign to potentially scale it up with your full 60-80 hours you'll be spending.
At some point you’re just draining yourself of every spare hour, you’re underperforming at your day job, and you’re not really moving forward.
It’s ok to do for a month or so as a litmus test, but after that I fully support diving in full time (in full knowledge that there’s a 95% chance you’ll fail).
You can always get another job.