I was in your exact situation. My day job at the time was very fulfilling. I only took the moonlighting gig for money. I hated it - it was a web design job and I'm a research/algorithms/systems person. I ended up spending way more than my billable hours procrastinating and stressing about the gig. My effective wage was shit. I didn't have free time to do stuff I enjoy.

I was young. I bailed hard and burned bridges. I could have handled it more professionally but I don't regret my choice. Business commitments and networking are not worth your sanity. If the gig is not essential to your career goals, you will never need the client's goodwill. Since you have a day job, you can omit it from your resume and nobody will ever know.

In the end, I probably spent all the extra money on drinks to ease stress and restaurant meals because I didn't have time to cook. Unless you need the moonlighting gig to survive, stick to your day job. Live frugally and try to build skills/experience that will help you earn more money in the future.