Could you provide a few examples from your workflow that requires you to open Xcode?
if you want to build ios apps installing xcode is a requirement (to get the proprietary cli tools) but launching xcode is not required
to build native mac apps, neither installing nor running xcode is necessary, as you can build with the smaller cli-only tools which include only mac os headers/libs