The downsides? Well, I find some of its syntax choices a bit odd and inconsistent; I think the pursuit of elegance could have gone much further. And, while I celebrate its pervasive use of closures/lambdas and functional idioms in my many contexts, it lacks true functional data structures that would be necessary to really write in a proper functional style with confidence that the runtime wouldn't suffer.
Also, the implementation of its high-level features is somewhat magical and hidden. You can't depend on performance when using them until you really study them. So much so, that WWDC had a big talk about how to choose language features depending on the type of performance you need, and none of it was intuitive.
Additionally, at least on the Apple platforms, its Objective-C heritage can be a bit annoying. In particular, you can't mix and match closures with "selectors", which is a real shame and missed opportunity for flexibility and elegance.