Hacker News Reader

Comment by gpderetta | Hacker News Reader

As a exclusive emacs user for the last 20 years, I'm quite excited; my main complain about emacs is its slow down with some more sophisticated packages. I wonder if it improves magit performance with large codebases.

Pigs fly just fine with with enough thrust.

View on HN

One reason magit is slow (that last I checked still hasn't been fixed) is that it spawns a large number of processes calling out to git for each operation. Most of these are redundant and could/should go away with either a redesign or a working caching scheme. This issue is more than obvious on platforms (e.g. some macOS versions) where fork/vfork are not as fast as one would expect. I like the paradigm behind magit, but the implementation leaves a lot to be desired. So the pig in this case is definitely not Emacs.

Example:

  # Doing a simple magit refresh results in the following processes being spawned (tracked with dtrace)
  # Notice how many calls are completely redundant

  2020 May  4 12:46:22 11819 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11820 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11821 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11822 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11823 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11824 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11825 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11826 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11827 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11828 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11829 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11830 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11831 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11832 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11833 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11834 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11835 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11836 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11837 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11838 <67471> 64b  /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...>
  2020 May  4 12:46:22 11839 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11840 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
  2020 May  4 12:46:22 11841 <67471> 64b  /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...>
  2020 May  4 12:46:22 11842 <67471> 64b  /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...>
  2020 May  4 12:46:22 11843 <67471> 64b  /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...>
  2020 May  4 12:46:22 11844 <67471> 64b  /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...>
  2020 May  4 12:46:22 11845 <67471> 64b  /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...>
  2020 May  4 12:46:22 11846 <67471> 64b  /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>

Magit is working on switching to libgit2 https://github.com/magit/magit/issues/2959 to fix this.

This issue has been open since 2017 and it's not panacea since it comes with its own -substantial- tradeoffs which is why as far as I know, magit is not "switching to libgit2" but planning to offer libgit2 as an additional option. Also for such an important feature, rather than going with FFI, it would be much better if Emacs linked with libgit2 (similarly to the improved JSON support in 27). That way the core Emacs development team would review all code and have a say in its design. That also implies magit being bundled with Emacs (or at the very least available in ELPA) which may not be realistic in terms of copyright assignment / code quality concerns.

Performance issues aside, a pure Lisp magit is the simplest and more universal setup which is why an attempt to fix the process spawning problems there should be made.

Why do I want the Emacs team reviewing libgit2 or Magit?

They would be reviewing the C interface to libgit2, not Magit itself, since -you know this- C code written either to work through FFI or as part of Emacs can potentially corrupt memory and destabilize Emacs in serious ways. Code written in Emacs Lisp, even low quality code, does not generally suffer from these issues.

So yes, I'd much prefer C code that Daniel Colascione / John Wiegley / Eli Zaretskii / Stefan / Paul .. have reviewed over newly-written FFI code that hasn't been through that thresher. Everyone that jumped on the emacs-libvterm train early knows what I mean.

I keep Emacs running for months at a time and it's the foundation of pretty much everything I do on a computer. Other than the OS, it generally is the most stable, continuously running, continuously stressed piece of software that I have ever used.

that's probably one reason, but as far as I can tell, a large cost is formatting an colouring the master diff buffer when there are a lot of changes. When doing very large merges, it takes forever to refresh the buffer but aborting whatever magit is doing will leave an uncoloured but perfectly usable buffer.

Also whenever somebody accidentally commits or stages node_modules. Seriously, had that happen to me before, and Magit choked on it.

You can somewhat fix the macOS Magit perf problem --- at least the Aquamacs one --- by patching the vfork in the source. Doing that took me from keystroke-lag speed to regular interactive speed (but I've since switched to railwaycat Emacs, which I don't think has the problem, though maybe I just don't notice it).

This may have just been that you moved to railway at emacs at the same time you upgraded past 26.1 which fixed the virology thing on macOS

Meaning, it's only really slow on macOS.

Spawning processes on macOS is annoyingly slow :( I think the Linux kernel developers have put a lot more time into making sure forks complete much more quickly.

Multiple versions of macOS. And Windows. And possibly other less known systems.

Besides, why would a discerning engineer put up with magit creating all these wasteful processes knowing that most of them are redundant, regardless of performance feel? If I was a magit developer, I would surely try to fix this.

On Windows it's not as fast as Linux, but it's not terrible either (as far as I heard).

Also see https://stackoverflow.com/a/16902730/615245 (I think it's not needed with the latest versions of Git anymore, though).

> If I was a magit developer, I would surely try to fix this.

Since you're just a regular user, did you check out the issue tracker for related discussions?

This issue is more obvious on windows where process creation is very slow.

Emacs 26.1 should perform much better at process creation as it uses vfork on macOS instead of fork.

I don’t think that caching is an easy solution. Git can do a lot of weird things so heuristics are bound to be unreliable, brittle, or both. Furthermore the wins from caching are small for the typical users who use systems that are capable of starting processes in good time.

Here's a caching patch that eliminates a few calls to git rev-parse per magit-diff/status etc operation that John Wiegley posted in one of the magit performance issues (modified to apply to recent magit): https://github.com/dandavison/magit/commit/08317454bf180d502...

Reply on news.ycombinator.com