No, you really don't. Apple would like you to use it (god knows why), but it's not necessary.
Is it possible for 100% of situations? I don't know, because I haven't tried 100% of the situations. And in one case I haven't figured out yet (AUv3).
You obviously still need to use signing pipeline etc, but that can all be done via CLI.
As for "non-trivial", here are Chrome's build instructions for macOS: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/m...
Which contains the words "While using Xcode is unsupported" :)