I can really relate to that. It's necessary to choose something that is not too far out of the comfort zone. Like choosing something novel is good but the majority of the libraries/tools should not be new. That's key to be able to both start and continue a long time. I'd make sure the first proof-of-concept can be done within a month, afterwards I'd either stop the project or continue. Otherwise it's just too frustrating.
Afterwards you can still build stuff on top. If the problem is open ended but has a reachable first plateau that provides a basic use case this can be a nice "rabbit hole project" that you can spend an infinite amount of time on if that's what you're looking for.