Some things need to get done, at least short term, sure. But if that's structural, some major chances should be done.
If a side project is neither a source of learning, nor fun, nor of opportunity (and thus "it doesn't pay off", in your words) - then just don't do it and find something else.
I'm a fierce believer in intrinsic motivation. Extrinsic motivations don't hold up. You're set to give up sooner or later.