Potentially dumb question, were you testing with optimizations enabled? ARC inserts calls everywhere, but then the optimizer does its best to take out redundant ones. It's not perfect, but it can make a big difference.
Good question actually... It was a few years ago; I know I was eyeing through my code using Instruments but I'm not sure if I profiled with optimizations on. I'll have to look back at my code to see if it'd make an impact with them on. Although having your debug build suffer significantly could be a valid concern.
So I took out all the "__unsafe_unretained"'s I inserted and indeed, with optimizations enabled in my release mode, the calls were removed. However, my no-optimization debug build is unbearably unusable like this so I cannot justify making this change.