Hacker News Reader

Comment by war1025 | Hacker News Reader

At my work we use sparse checkout and lfs on our binary dependencies submodule to pull in only the binaries that we need for the current platform (i.e. linux or windows)

Basically sparse checkout only populates the tree for the dependencies we want, and then git-lfs will only download the binaries that are present in the current worktree.

Works out pretty well.

Keep in mind though that sparse checkout still has the entire repository loaded in the `.git` object store, it just doesn't expose it in the worktree.

View on HN

Are you sure about that?

> This combination speeds up the data transfer process since you don’t need every reachable Git object, and instead, can download only those you need to populate your cone of the working directory

If you're only downloading what you need to populate the working directory how is it that `.git` will have the entire repository?

I guess it's possible you don't get it all, but I've definitely ran `git grep` before on that repo and had results come back that weren't in my worktree.

Edit:

   wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ git grep "def returnValue"
   twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/install_linux_gcc54/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):
   twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/install_win64_vc141/lib/python2.7/site-packages/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):
   twisted/vendor_base/src/twisted/internet/defer.py:1350:def returnValue(val):
   twisted/vendor_base/src/twisted/internet/test/test_win32events.py:66:    def returnValueOccurred(self):


   wrowclif@wrowclif-desktop:~/Taccs2/p5_deps$ ls ./twisted/
   install_linux_gcc54

I would speculate that the partial-clone implementation pulls down all the commits that touch any files that are required. Some of these commits would presumably include changes to other parts of the source tree. Perhaps `git grep` still matches on such commits?

partial clone is different from sparse checkout.

We are using sparse checkout. Partial clone is the one that only pulls down objects that are needed by the store.

LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand.

Git’s partial clone is a more natural way of achieving the same outcome.

Using sparse-checkout by itself will still download the entire repository and its complete history into .git. If you additionally use the "partial clone" feature, then you can restrict what gets downloaded and stored in .git as well - it will download only the objects that are needed for your selected directories (along with their complete history). On big repositories with long history this might still be too much data, so you might also want to use the "shallow clone" feature (via the --depth flag) to restrict how much history you download.

Reply on news.ycombinator.com