A lot of it will also depend on your advisor. A more hands-off advisor will certainly mean trouble for you, as the required pressure to keep you focused will not be there. Finally, conference deadlines are the other big motivator for PhD students. If you don't think that would help focus you, then you'll be setting yourself up for failure (and possibly depression).
Also, I'm not sure how feasible it is to do a "part-time" PhD. That might work with a master's where it's almost exactly like bachelor's part 2, but a PhD is more of an all-or-nothing endeavor. The one exception I know of is for people starting/spinning-off companies based on their research. Even there, it's a lot of work, and I personally find it very hard to maintain good forward progress in both at the same time.