This is just speculation, but slowness might be because this is being run as a script via /usr/bin/swift, rather than being precompiled into a binary.
Certainly I have found my own Swift scripts seem to be a lot slower than compiled programs.
EDIT: from my own test with 1000 iterations, script version 17s, compiled version (release build) 5.4s