Hacker News Reader

Comment by MayeulC | Hacker News Reader

Interesting. I've been wanting something like that for submodules. Can the two features be combined?

For instance, if you need a single file/directory from another project in your repository.

I thought one of the big benefits of monorepos was that you didn't mess with submodules anymore?

It still might be needed for external dependencies. the code an organization writes might be in one repo, but if you want to bring in some other library, like libssl (assuming there is no better package manager for your language) submodules are often used

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows)

Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree.

Works out pretty well.

Keep in mind though that sparse checkout still has the entire repository loaded in the `.git` object store, it just doesn't expose it in the worktree.

Are you sure about that?

> This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory

If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree.

Edit:

   wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue"
   twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):
   twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):
   twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):


   wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/
   install_linux_gcc54

I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?

partial clone is different from sparse checkout.

We are using sparse checkout. Partial clone is the one that only pulls down objects that are needed by the store.

LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand.

Git’s partial clone is a more natural way of achieving the same outcome.

Using sparse-checkout by itself will still download the entire repository and its complete history into .git. If you additionally use the "partial clone" feature, then you can restrict what gets downloaded and stored in .git as well - it will download only the objects that are needed for your selected directories (along with their complete history). On big repositories with long history this might still be too much data, so you might also want to use the "shallow clone" feature (via the --depth flag) to restrict how much history you download.

> For instance, if you need a single file/directory from another project in your repository.

The last time this happened to me, I took it as a hint that I had split the repositories along the wrong lines. The repos should probably be either merged or divided further to prevent this.

Sometimes you don't own the other repo.

Doesn't that seem like a build tool situation? At that point the other piece of code isn't part of source, it's a source dependency, and no different from a binary dependency at some version so you don't really want the tree, you want the file at some revision and if it's `github` based then you have the natural HTTP endpoint and otherwise it's trivial to proxy as an artifact.

Not really. You often need to make extensive changes in those kinds of external dependencies, so you really do want them in your source tree.

Oh, interesting. That explains why you want to retain the history, so you can easily merge and stuff too.

Well, you are right, but there would still be some advantages to submodules:

1. Check the files hash themselves: while you can definitely put the commit ID in the URL, nothing prevents the remote server (though unlikely if github) to answer with another version of the file (and could even do so selectively for your build server).

2. Simple upgrade path: with submodules, you can just `cd` into them and run `git pull` or `git checkout v11.5.2`, and git itself could inform you that a newer version is available if tracking a branch.

I also agree with the contribution aspect, though it is less important in some cases.

I take the latest example I have in mind where this could have been useful: For integration into F-Droid, RiotX needed not to include binary artifacts of a library, but the source itself. The source repository is quite big (multiple languages), but the thing of interest is a single java file [1]. They ended up simply copy-pasting the file [2] in their repo, which makes its origin less obvious, and more subject to bit-rot and vulnerabilities.

[1]: https://github.com/google/diff-match-patch/blob/master/java/...

[2]: https://github.com/vector-im/riotX-android/pull/760

Reply on news.ycombinator.com