The defacto standard matters. The vast majority of Swift programmers will be using Xcode, and I definitely immediately think of the pains I've had with Xcode when I hear Swift. For example, I don't know of any other good IDE environment for Swift, though maybe there is one.

You could also argue that e.g. Rust is not cargo, but almost every Rust programmer will be using cargo. Sure I could use something else, but why would I? Of course the analogy is not perfect, because the "why would I?" is clear for Xcode: it's bad and macOS only :D

All I'm trying to say is: defaults matter. Most people will not be writing Swift in VS Code (on macOS).