In my experience, it depends a lot on the sort of libraries you need. The library ecosystem is still relatively small and client centric. Swift can be a fast language (if you avoid reference counting) but many libraries are incredibly slow and clearly unsuitable for server-side use.
For instance, I recently needed reasonably fast JSON and CSV parsers with predictable memory usage for incrementally parsing potentially large files. The libraries I found were either too slow or clearly written for client side use cases like mapping a few kilobytes of config files to objects in memory. I ended up writing my own, which was absolutely not what I wanted to spend my time on.
The upside is that Swift's C interop is excellent and now it's even getting C++ interop. So one way around library issues is using Swift like Python, as a thin layer on top of C. But as this FoundationDB talk shows, no one is really happy with this state of affairs.
I think server-side and cross-platform Swift is moving ahead at pace. Things will be looking lot brighter in two or three years.