Come on now, you're going to need Xcode at some point during development.
Is it possible for 100% of situations? I don't know, because I haven't tried 100% of the situations. And in one case I haven't figured out yet (AUv3).
You obviously still need to use signing pipeline etc, but that can all be done via CLI.
As for "non-trivial", here are Chrome's build instructions for macOS: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/m...
Which contains the words "While using Xcode is unsupported" :)
So far I’ve been shipping internal apps without it! Even notarizing works :)
Apparently Audio Units v3 is only available with Xcode, which is why I’m only doing v2.