stolee

Not deprecated. You can still use it. You need to manually opt-in using `git config core.sparseCheckoutCone false` since the default has switched.

If you are curious, then you can find the discussion starting at v2 here: https://lore.kernel.org/git/20220330190956.21447-1-garrit@sl...

This is a first time contributor getting a headlining feature at the top of the release notes!

This is a really core and famous problem in extremal combinatorics and a lot of people think about this problem and many related ones. I think the biggest impediment is that too many people were looking for the upper bound instead of looking for a lower bound. The construction here is still incredibly novel and requires sophisticated mathematics to prove it is correct.

Yes! Also cherry-pick and revert.

But specifically, the ORT strategy can cache computed renames across multiple commits being rebased, so the performance benefits are even greater for "git rebase".

That history that looks tangled and awful would look a lot better if the commits were sorted by `--topo-order` instead of `--date-order`. That sort “groups” commits that are in a single line of history.

You're right that we used what we learned from Scalar's background maintenance an applied that to Git itself.

Putting background maintenance into Git was actually part of our effort to get Scalar on Linux.

You might be interested in our "Philosophy of Scalar" [1] document, which includes this paragraph:

> Scalar intends to do very little more than the standard Git client. We actively implement new features into Git instead of Scalar, then update Scalar only to configure those new settings. In particular, we are porting features like background maintenance to Git to make Scalar simpler and make Git more powerful.

[1] https://github.com/microsoft/scalar/blob/main/docs/philosoph...

The delta compression in Git is about storing the file contents of an object as a diff against another object. This changes the literal size on-disk, but it doesn't change the logical unit.

In fact, the delta chains used by Git for space compression have no direct relation to the object model DAG. From the perspective of a user using Git, these deltas are completely invisible.

Edit: perhaps to help this point... If Git stores an object using a delta, that doesn't change the object ID of that object compared to storing it uncompressed.

If I have a 100-line file and on 'main' it changes near the top, but in my 'topic' branch it changes near the bottom, then I can cherry-pick 'topic' onto 'main' and Git will resolve the diff correctly. The resulting diff or patch would only change in the line numbers for the context of the diff.

This is of course a very simple example. You might hit a conflict in your "git cherry-pick" command which gives you an opportunity to resolve the unexpected diff issue in an appropriate way, which ends up with a different diff than before.

We fixed the typo in the blog title (shapshots). Sorry!

That operation is much faster in recent versions of Git, especially after a GC or “git commit-graph write” command.

If you don't disable fetch.writeCommitGraph and gc.writeCommitGraph there is a chance that those operations will overwrite your commit-graph and delete the changed-path filter data. (fetch.writeCommitGraph uses the --split option so it might not actually erase the data until you have accumulated enough "new" commits)

You don't need to rewrite the commit-graph file every time you want to run "git log". The "git log" command will parse the newer commits the old-fashioned way until you reach the commits encoded in the commit-graph file. If you do it once now, then you'll still be fast even if a few commits are added on top of your existing history.

If you update the commit-graph with that command once a week, then you'll stay fast even in a very large repository.

The data is stored in your `commit-graph` file by this command, but those other ways to update the `commit-graph` file don't pay attention to the fact that the data exists in order to persist it.

There is work in progress to fix that issue, hopefully in the next version: https://lore.kernel.org/git/f1e3a8516ebd58b283166a5374843f5c...

Thanks for the link to the documentation. That is updated with every major Git version, and can be used to track what features are present. Also, the release notes can be helpful.

In particular, the recent Git v2.27.0 release does include an implementation of Bloom filters with speedups for `git log` and `git blame`. You need to manually run the command to make it work. The version I prefer is this:

> git commit-graph write --changed-paths --reachable

After that first write (which writes filters for every reachable commit) you can do a smaller write by adding `--split` to write incrementally [2]

[2] https://devblogs.microsoft.com/devops/updates-to-the-git-com...

By writing these filters, you will speed up most `git log` and `git blame` calls. There is an improvement coming in the next version that includes speedups for `git log -L`.

Caveat: The biggest reason these improvements have not been widely advertised is that the user experience has not been completely smoothed out. In particular, you can only write the changed-path Bloom filters using the command(s) above. If a commit-graph is written during GC (due to `gc.writeCommitGraph` config setting) then the filters will disappear. Similar for `fetch.writeCommitGraph`. We plan to have these resolved in time for v2.28.0, along with more performance improvements.

(Full disclosure: I am a contributor to Git, Scalar, and VFS for Git, which are referenced by the article.)

LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand.

Git’s partial clone is a more natural way of achieving the same outcome.

The sparse-checkout patterns match at the file level, so you can always use that (without “cone mode”) if you want. It becomes difficult to match an exact file list as people add files to projects: you require every other user to update their patterns to match the newly-added file.

The comparison to hill-climbing algorithms is an interesting one, especially because the author ignores the fact that hill-climbing algorithms get stuck in local optima unless there is a way to “restart” the search from a new location. There is a time and place for big, risky changes to advance beyond what this “patch” approach can do.

Earlier this year, we considered similar changes to the ahead/behind calculation in 'git status'. We concluded that there is no good way to provide partial information. Specifically, you can't say "10 ahead" without walking everything you are behind to find all merge-bases. That "10" you found could become smaller by walking more of the other set. We instead opted to not calculate that information as often.

Here is my reply to the thread on-list that summarizes why I think this direction is futile: https://public-inbox.org/git/20180108154822.54829-1-git@jeff...

This ahead/behind calculation is in a lot of places, including 'git fetch' where it checks if each ref update was a forced update (checks if the new ref value has the old ref value in its history). For our version of Git that ships with GVFS, we added an option to skip this check, providing a significant speedup to users fetch times: https://github.com/Microsoft/git/commit/9616c7da3141f539a425...

In your calculation, you are assuming that all of those files are in independent parts of the folder hierarchy. In practice, deep directory structures mean that single commits are likely focused in a few very deep folders. We ran the statistics on a number of repos with different shapes to come up with this number, and it covers 99% of commits that we measured.

Do you mean `git branch --contains` or `git branch -vv`? There are many options to `git branch` that cause Git to be very slow.

The commit-graph feature in general will make these faster by reducing time spent parsing commits. You can compute a commit-graph right now if you have Git 2.18 installed: https://blogs.msdn.microsoft.com/devops/2018/06/25/superchar...

Generation numbers will make these operations much faster in Git 2.19. Here are the related commits:

`git branch --contains`: https://github.com/git/git/commit/f9b8908b85247ef60001a683c2...

`git tag --contains`: https://github.com/git/git/commit/819807b33f820dc17d96f04374...

It does! The Bloom filter stores every path that changed, including paths to trees (except the root tree, which is expected to be changed by default). When you change a file, you also change every tree above it.

The good news is that if you are using shallow clones, then you probably don’t have enough commits locally to need the commit-graph feature!

We are working to make this run automatically in the future. https://public-inbox.org/git/20180627132447.142473-1-dstolee...

You don't need to run this with every commit, but maybe once after a big fetch or right after a new clone.

Sorry for the worrying note about the experimental feature. One issue when working in open source is that contributors don't have control over the release cycle, and review requires smaller series than having the feature be delivered all at once.

These interactions with grafts, replace-objects, and shallow clones are one reason 2.18 does not create and manage this file automatically. The commit-graph file works by representing the commit relationships in a new file, and if that file exists, we treat that as the truth. The commit grafts, replace-objects, and shallow clones use another set of special files to track commits whose parents have been modified in special ways. If you would like to see our progress on integrating these features together, please see this thread on the Git mailing list: https://public-inbox.org/git/20180531174024.124488-1-dstolee...

Your second question about "thousands of commits an hour" is actually a server scale question, as opposed to a GVFS question. The main difference is that GVFS is primarily client software plus some extra server protocols that a server must support.

I find the "thousands of commits an hour" measurement to be conflating multiple things. I'll split out my expectations in our experience with the Windows repo.

The Windows repository is hosted in VSTS and handles over 4,000 developers doing their daily work. This includes an average of 3,000+ pushes per day via the Git protocol. Each push may include multiple commits. At least 2,300+ pull requests are completed per day, which creates a new merge commit each time. Also, the build machines run daily and kick off a push as their first action to update a version number. This creates 250+ parallel pushes within a 10 second window, so a spike of concurrent pushes are supported by VSTS.

There isn't anything wrong with the objects. A 'fetch' succeeds but the 'checkout' is what blows up.