Were your performance problems before or after Swift 5's switch to using UTF-8 as the preferred encoding? See https://swift.org/blog/utf8-string/ . It sounds like before, so you may want to check out the String performance again.

Swift 5.0 and 5.1 each supposedly made some 10-20% gain in ARC performance compared to the previous version (4.2 and 5.0), so that may also help. I think Swift's performance can be very hit or miss but they (Apple) are making some good strides.