Based on my experience I can say that, persistence is what matters. If you work full-time and are not persistent or loose focus then you are not going anywhere. If you work part-time and are persistent and focussed then you will win. May be it will take more time than usual but you will get there with better results.
So I would say, yes go ahead and work part-time on your idea, but be very persistent to spend whatever extra time you find on the idea. It would be good idea to find group of friends who can work part-time with you. (I did that it does help a lot)