I did some further investigation [1] by running 20 million iterations in Release and profiled the code.
Turns out, at least 82.9% of it was just swift_retain / swift_release (ref counting). Whether those retains / released can be optimised away, I don't know, we'll have to wait and see.