I don't think I've ever seen anyone regard the compiler as tooling, SourceKit for sure that makes sense, but not the compiler. Especially in this case as there really is just the one compiler for it so those are issues that will hit anyone using Swift now. But if you want problems with the language itself, let me see:

It suffers from similar problems to Scala, where it tries to blend OO concepts in with FP concepts. Variance is where this flares up terribly because it doesn't provide any explicit support for covariance or contravariance. You end up with invariance and subtyping in a bunch of places which is not a nice combination.

Protocols like Equatable and Hashable are implemented with compiler magic for arrays and tuples. Which means that you run into trouble with generic functions that say accept two instances of the same type which is Equatable if you pass an array of ints. The underlying reasons behind this (which mostly escape me at midnight on a Sunday) are something to do with protocols on generic types. You can't say "for a List if the elements are Equatable the List is equatable" IIRC. This one annoys me no end as we've had to fudge our way around it several times on the same project.

For some reason that eludes me, Optional<T> is given all sorts of special case syntax ("if let" and "guard let") which is like a crap version of "for yield" from Scala or do notation in Haskell that only works on that. As a result a lot of convenient abstractions are just not possible you end up writing all kinds of horrid looking chained map/flatMap calls instead. Sometimes because of this dissonance between things a block of code might have a guard let block, then some other stuff, then an if let block with an else condition. Whereas if you wrote the same thing in Haskell it would be one do block and that's it.

Concurrency and parallelism support is effectively Grand Central Dispatch, which is the most imperative API ever on macOS/iOS and on other platforms (according to the Github page I just looked at for it) is in the early stages of development. You're just calling the old Objective-C API and the language doesn't help at all there.

Edit: As a bonus addition, it's a statically typed "FP" language which doesn't have higher kinded types which means a bunch of handy abstractions are a real pain. See the Swiftz project for how they have to define things like monads as an example.

Is that enough problems?