1. Swift's ARC uses atomic reference counting underneath, which is normally very expensive, and relies on compiler optimization to remove as many reference count operations as possible. This is normally pretty effective, but there are situations where it's not possible.
2. Reference counting allows for arbitrary long pauses as the result of cascading deletions (i.e. where object deletions trigger other object deletions). You can work around that (by deferring deletions), but then you don't have any guarantees about the timeliness of deletions anymore. As far as I know, this is still an open issue for Swift.
3. Without a compaction scheme, you risk memory fragmentation. While this is a rare occurrence in practice, there are workloads where it can happen.
4. Reference counting cannot reclaim cycles without a mechanism for detecting cycles; such a cycle detector (e.g. trial deletion) poses pretty much the same challenges as tracing GC.
Obviously, tracing garbage collectors pose their own challenges; my point is merely that whether performance and memory usage are more consistent has to be judged on a case by case basis.