I have recently been developing a server-side Swift application, a few months of sporadic evenings and weekends. My starting point was having a year or so of professional iOS experience and a ton of web/server/infra experience.
My experience with it has certainly been mixed, but in many ways a breath of fresh air compared to some web development ecosystems. At work I'm mostly using Java on the backend and I think Swift could be a really nice replacement for that for larger systems – the type system is better, the ergonomics are better, the developer tooling is much better at the lower end (although there's plenty of specialist JVM tooling that is better).
Ecosystem-wise, Swift has few but high quality libraries available in general. Apple's first party libraries are exceptional. Vapor is in a weird place though. Some parts are really mature and well considered for their relatively young age (e.g. parts of the ORM, async stack), others feel very early days for things that have been around for a number of years now (e.g. queues). Others feel like an impedance mismatch in the framework design – I find the Content system both too limiting and not providing enough.
Overall, I'm loving it for a personal project. Would I use it as the foundation of a new startup? Probably not. Would I use it in a iOS heavy company? Yeah probably. I think you can do productive, high-quality backend engineering in Swift, today.