Seems like performance always works in Swift's favor when the compiler is able to reason about the types (e.g. no AnyObject nonsense, banishing the objc dynamic behavior).
Interesting to see Generics performing so poorly, though.
I'm guessing the use of AnyObject is implicitly bridging Int and String to their non-struct types. The benchmark for an array with type Any seems to perform twice as fast as AnyObject.