If you're not developing an iOS/macOS app, you can skip Xcode completely and just use the `swift` CLI, which is perfectly cromulent. (It works great on Linux and Windows.)
If you hate Xcode, this is definitely worth a look.
That's a pretty good deal.