Maybe the reason is another? Personally, I once had Magit act real slow at times because git-annex had added some git hook for something that I didn't need. Just removing that hook improved performance massively. Perhaps something similar is happening.
Also, I'm not sure Emacs Lisp can interface directly with C libraries, so using libgit2 might not be an option.