You can install a Swift toolchain on macOS without Xcode, but Xcode is the “blessed” route and listed first on the swift.org getting started instructions.
https://www.swift.org/install/macos
However, it can be a bumpy ride. For example: “swift test” (via the CLI) doesn’t work if Xcode is not installed. https://github.com/swiftlang/swift-package-manager/issues/43...
I would just emphasise that the OP was about _convenience_ and Swift-without-Xcode on macOS is not smooth sailing.