You can install Swift on Linux if you want and code away, just like with Rust - but as it hasn't really caught on for anything other than building apps for the Apple ecosystem, it's not a particularly normal thing to do.
However, it can be a bumpy ride. For example: “swift test” (via the CLI) doesn’t work if Xcode is not installed. https://github.com/swiftlang/swift-package-manager/issues/43...
I would just emphasise that the OP was about _convenience_ and Swift-without-Xcode on macOS is not smooth sailing.
Even if you don’t have Homebrew, the first time you try to run `swift` in a Terminal you’ll get a GUI prompt which lets you install the Developer Tools in two clicks.
Or you can run `xcode-select --install` (comes preinstalled, no Xcode needed).
You could also argue that e.g. Rust is not cargo, but almost every Rust programmer will be using cargo. Sure I could use something else, but why would I? Of course the analogy is not perfect, because the "why would I?" is clear for Xcode: it's bad and macOS only :D
All I'm trying to say is: defaults matter. Most people will not be writing Swift in VS Code (on macOS).
Equalling Xcode and Cargo makes no sense. A similar situation to Xcode and Swift would be like Visual Studio and C# on Windows. Many developers use these tools, but they are not the language ecosystem, and the Cargo equivalents are entirely separate (Nuget, Cocoapods).
I'm still really sad Jetbrains decided to sunset AppCode :(
So the quality of tooling/IDEs is definitely a factor, I just don't see myself using Swift outside of the Apple ecosystem when there are so many other alternatives.