That's actually a git issue - git is notoriously slow on OS X, and magit uses git heavily in a way that is correspondingly slow.