i am struggling with this too. one thing i am worried is that the sideproject will consume me and let me neglect other work that i need to do. in a way that is my fear of failure, not failing to complete the project but failing to give it the appropriate priority.
so this is one thing to look at: is there something that you are afraid of that might happen when you engage in this side-project.
find out why that is, and then try to address that fear.
if you are afraid that you will spend to much time on it, maybe schedule your time. only one hour a day, or only on one evening a week, or saturday afternoon, or only when you get to a particular location. in the Beijing GNU/Linux User Group, i started a monthly "Coding for Fun" event, where we'd meet to hack on our projects. a perfect space for a side-project.
if you are afraid of working alone on something, find a hacker/geek/linux user group that engages in related activities and find someone to join you on that project. again, our Coding for Fun afternoon was a perfect venue for that.
are you afraid of not completing the project, maybe try changing your goals, so that completing it is not the goal, instead have a goal that can be achieved even with a partially completed project.
another one of my fears is that the project is of no practical use to anyone. so i search for that practical use.
for example i have been wanting to learn smalltalk for years. but without a practical use, i could not do it. i'd download pharo images and then not get around to doing anything with them. in the meantime, i discovered that recreational coding is in itself a practical use. an hour of dabbling around, trying out something, learning something just for fun, without a customer being on my case, or without the pressure to get it right.
i now set up a pharo smalltalk environment so that i am be able to use it every day for a mundane task (it runs a commandline terminal), and now, whenever the mood strikes i can go off on a tangent and play. but i am not afraid of it consuming me (see above) because i'll still get to use it every day even if i don't go off to explore. compare that to learning linux by simply using it to read your email and browse the web without doing any focused learning activity, learn by getting used to having it around, then slowly find new activites. i can use smalltalk as a calculator for example. (i used to fire up pike or python for that before)
to bring that back to your project. find one good reason to work on that project, and as you go you may discover more reasons that help you to continue working on it.