I think you’re right. There is a heck of a lot to be said for enjoyment regardless of productivity.
It becomes problematic, for me at least, when the thing i enjoyed becomes a sort of default mode. No longer for enjoyment, more of a reflex. Then it feels like it might be wasted.