In my experience with Magit, the slowness usually comes from a large number of subprocess calls that are made to render some aggregated piece of information. Eg. some table that has a bunch of commit hashes, but then the way it's rendered, the hashes are links that need to have, as part of the link, some information embedded in them that's only available through a separate subprocess call, one per commit hash... Emacs Lisp obviously adds some overhead, but the larger problem is the need to make these dozens of subprocess calls.
I don't think there's an immediate solution to this problem though. Most likely, in specific cases where this amplification of subprocess calls Tartius (or whoever mans the project today) needs to rethink the buffer layout, make things load lazily, maybe add some caching... but this isn't going to happen quickly. On the bright side, these changes do happen every now and then. In the end of the day, I use Magit almost exclusively today, I even have my extensions to Magit (specifically for Git grep command), and I wouldn't trade it for any other Git interface. I wish it was better, but I haven't found anything that would've been better in practical terms.