I don't think Swift's performance is due to start up time at all. I actually cloned the repo, and ran the benchmark and found that Swift's execution time scales drastically with the size of the input. The Swift team actually boasts about its quick start up time on the official website [1]. I ran a simple hello world benchmark to gauge Swift's start up time and got an output of 13 milliseconds.
echo 'print("hello world")' > hello.swift && swiftc hello.swift -O -o hello
time ./hello