Hacker News Reader

Comment by iliekcomputers | Hacker News Reader

Nice work, really interesting blog post!

On a sidenote, git itself can also get painfully slow with large monorepos. Hope GitHub can push some changes there as well.

I know FB moved off git to mercurial because of performance issues.

View on HN

My understanding is that neither Git nor Mercurial can do this well out of the box, and FB and Google both have their own extensions to Mercurial to make this possible (because even though Mercurial is often slower than Git, it’s extensible)

e.g. https://facebook.github.io/watchman/ - used as part of Facebook’s Mercurial solution, I think.

Git also has a file system monitor interface which can use Watchman. We (GitHub) are working on a native file system monitor implementation in addition - https://github.com/gitgitgadget/git/pull/900.

And then from mercurial extensions to our own server, mononoke, which apparently has been moved under the Eden umbrella: https://github.com/facebookexperimental/eden

I thought Google used some custom fork of perforce.

From what I understand, Piper is not a fork of Perforce, but instead a completely different system with the same interface. You know, built on top of a BigTable or Spanner cluster instead of whatever Perforce uses.

The Mercurial extensions are then an alternative client for Piper.

It's my understanding that it started as a fork of perforce, Google bought a source license for perforce. Whether you'd still call a fork of perforce anymore is like asking if a Computer is the same computer after you've replaced all the components over the years.

That was the case many years ago (extremely customized perforce server), but after that there was a full rewrite as a distributed system built on Bigtable etc. - https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... is an overview. It wasn't a ship-of-Theseus type situation where they incrementally replaced components.

You might be interested in scalar [1] developed by Microsoft for handling large repos.

[1]: https://github.com/microsoft/scalar

It's also interesting to note how much of Microsoft's work for handling large repos in git has merged upstream directly into git itself.

One very interesting part of that is the effort that has gone into the git commit-graph: https://git-scm.com/docs/commit-graph.

It's part of what makes scalar interesting compared to some of the projects you hear mentioned used inside the FB and Google gates: not only is scalar itself open source, but a lot of what scalar does is tune configuration flags to turn on optional git features such as the commit-graph, sparse checkout "cones", etc that are all themselves directly supported by the git client. Even if you aren't at the scale where it makes sense to use all of the tools that scalar provides, you can get some interesting baby steps by following scalar's "advice" on git configuration.

If you are willing to adapt to a different structure and workflow, you can filter the scope of git down dramatically with sparse checkouts (as @WorldMaker also mentioned).

https://github.blog/2020-01-17-bring-your-monorepo-down-to-s...

Sparse-checkouts are amazing. I wrote some small tools that use dependency information in Flutter packages to drive a sparse-checkout. We use it at $dayjob now.

Google's approach to that is to make submodules better, so you can have a "superproject" which has submodules of multiple repos that looks like a monorepo. If you look at https://android.googlesource.com/ all the projects named as "superproject" are for this purpose.

Or at least that was the plan at 2016 when I left Google.

Reply on news.ycombinator.com