get/set/didSet introduces ridiculous complications just so we can pretend that a large number of function invocations look like assignment statements. This was a cool idea in C++, briefly, and then we discovered that having arbitrary code hide behind what looks like an ordinary assignment statement was a bad idea.
Extensions also remind me of C++ excesses. It is difficult to look at a piece of code involving a variable some type and be sure that I understand how that type works. The problem is that the code implementing the type is not in one place. There is the type definition, but then there are extensions. Possibly many of them. Defined all over the place.
I really dislike the irregularity surrounding let. In one context in unwraps an Optional. In another context it does not. Why?
I got off the Swift train at the release in which associated types were introduced. I'm sure it solves some actual problem, but it sure wasn't clear at the time. (The project ended -- it's not that I decided to go back to Objective C.)
Reference counting is far too intrusive, and it is easy to get it wrong in subtle ways. I find Java's (!) memory model relatively simple, and that's saying something. I understand why reference counting was chosen over GC. I don't have a good answer, but reference counting is just ugly. I actually think I prefer the C/C++ model of explicitly freeing memory.