I think sometimes people forget that Swift is still a relatively new (late 2014, ~7 years old) language. Given its scope and sizable feature set, it's no surprise there are many kinks to work through. Fortunately the team has been cruising and it gets more solid every year.
Right now it's a de facto for iOS and there's a nascent but passionate server community, give it a few more years and I think it'll be a much more attractive for things outside of iOS.