I was more thinking of smaller teams or individual developers that are Apple only.

If you have a bigger team, then it doesn't matter as much, because you have different people who do the iOS app, people who do the website, people who do the back end etc.

Swift on the server doesn't have to go mainstream to be useful. If you just need some basic web service that syncs highscores or verifies in-app purchases then it would be neat if you could write that in the same language as the app itself.