It's hard to say without an explicit goal. The most obvious general answer is: work on it when you feel like working on it, and don't when you don't. If you want to make it your primary focus, you have to consider: when do you want it to be your primary focus by? How much effort would it take to get the project to a state that it would be viable as your primary focus? How much time do you have to allocate to the project to reach that total effort before the time you want it to be your primary focus by? How long will you be able to enjoy this project being your primary focus, compared to how much time you spent not relaxing and enjoying your personal time? Will you even enjoy it if you can make it your primary focus, if you don't enjoy working on it in your free time now?
These are all questions you must answer for yourself in order to figure out how to set boundaries in a healthy way.