I’m pretty sure you need Xcode to compile Swift on macOS.
However, it can be a bumpy ride. For example: “swift test” (via the CLI) doesn’t work if Xcode is not installed. https://github.com/swiftlang/swift-package-manager/issues/43...
I would just emphasise that the OP was about _convenience_ and Swift-without-Xcode on macOS is not smooth sailing.
Even if you don’t have Homebrew, the first time you try to run `swift` in a Terminal you’ll get a GUI prompt which lets you install the Developer Tools in two clicks.
Or you can run `xcode-select --install` (comes preinstalled, no Xcode needed).