I believe Swift occasionally has problems with type inference. If you time your builds and functions then you'll see where you need to add an explicit type.
I've also noticed image and color Assets take a while for an initial build.
For example, this little app of mine takes 25 seconds for a full build, and less than 1 second for an incremental, probably due to the 100 pictures.