Yes, I have been doing it on a four year old, Core Duo one for the whole three years since the iOS SDK became available and have had zero compilation performance issues. Keep in mind that the vast majority of iOS applications are laughably small compared to desktop apps.
I'm not disputing that a Mini is still too costly for many, but I think it is unfair to exclude it as development machine.