If you’re spending 40+ hours a week at work and not being able to “focus on your technical skills”, you’re doing it wrong.
Besides, in my experience, knowing a specific technology from doing side projects usually doesn’t count as much to potential employees as having on the job experience unless your open source project is really popular.