How to handle big repositories with Git | Hacker News Reader

How to handle big repositories with Git

blogs.atlassian.com

143 pointsmmastrac33 comments

I wrote a Git extension about a year ago that transparently stores data in S3 / Cloudfiles / etc. and doesn't store any of the actual data in your Git repo. I've used it with a few projects but I think it could be battle tested a bit more. It integrates perfectly with GitHub / Bitbucket. Pull requests welcome!

https://github.com/lionheart/git-bigstore

Version control is not enough for binary files. Binary files need access control since changes can not be merged. Older centralized version control often provides access control as well. Distributed version control can't do this by it's very nature

I'm not sure why this ever got downvoted, because it's a key point. Unless you've got some kind of exclusive check out/check in type facility, working with unmergeable binary files is going to prove a pain. It's no fun having people find out once they come to commit/check in/merge/etc. that they've got the unenviable choice of losing their work, or losing somebody else's... or losing both.

(The disk space problem is far less of an issue - disk space is cheap, and networks are pretty fast, so up to some medium repository size of (I suppose) hundreds of GB you can actually just let it build up. This doesn't scale completely, but for many projects, possibly even most of them, if everybody has a copy of absolutely everything, it's no problem.)

For whatever reason DVCS users often have a bit of a blind spot in this regard. Because DVCSes are good, the thinking often seems to run, if DVCSes handle something poorly, that must be something that revision control simply isn't to be used for. ("You must be using it wrong. Try storing everything on a shared drive and take daily backups and coordinate access using physical tokens.") Which is wrong, because revision control is for everybody, not just the programmers. People who work on unmergeable binary files need it too.

But fortunately, even if git/etc. and their users doesn't care about you, you're not completely stuck. if you have lots of binary files your needs are fairly well served by Perforce, and you can probably get on OK with SVN and its locking facility.

This is an excellent point; I guess you can't drop the "connectivity to central server" requirement and still implement global locking.

Having said that, some DVCSes do support a form of locking - similarly to changes, tags, etc, locks can be created and pushed / pulled. An example would be Veracity's implementation:

http://veracity-scm.com/qa/questions/1105/how-do-file-locks-...

Obviously this is not quite as foolproof as with a centralised version control system, but using strict workflows (i.e. always pushing locks immediately and pulling before you work on binary files) could create a workable compromise. I suppose you could create some plugins for your favourite DVCS to emulate this behaviour, and to enforce the workflow.

PlasticSCM [1] offers a mixed approach to handle both distributed and centralized patterns for both large files and branch-happy repositories. It's worth checking out. Some mid-size game studios are using it successfully, and I plan on a serious testdrive as soon as I can catch a breath.

[1]: http://www.plasticscm.com/home.html

Like they say, there's an extension for that:

http://mercurial.selenic.com/wiki/LockExtension

Unfortunately by definition that's not a solution. Because hg is distributed checking out the file locally does nothing unless the fact that it's locked is propagated to everyone else's copy of the repo immediately and that doesn't happen.

By definition?

Did you even see what this does?

> Even though the bounds that identify a repository as massive are pretty high – for example the latest Linux kernel clocks at 15+ million lines of code

Yeah 15 million lines of code isn't a massive repository, it is medium-large at best. Any one of the big enterprise software companies has repos an order of magnitude bigger for each major product they sell.

This is very disappointing when it comes to dealing with large files.

It looks like git-bigfiles was abandoned years ago, after making very little progress, and neither bitbucket nor github seem to usefully support git-annex.

I have been quite happy with Mercurial's largefile extension, which has been part of core hg for quite some time:

http://mercurial.selenic.com/wiki/LargefilesExtension

I think this must be one of the reasons why hg has some popularity in gamedev.

Didn't Linus mention this in his talk at Google, Mercurial offers similar speeds, but Perforce would be the option for large binary files.

I personally haven't come across any need to use anything except shallow clones in large repos. Most of the time, you want to keep those other topic branches regardless.

They've linked to their previous most regarding submodule in the post[1], but it's worth re-mentioning that if you need to use submodule, you should almost always use subtree.

[1]: http://blogs.atlassian.com/2013/05/alternatives-to-git-submo...

I cannot tell if I am way off topic here, so pardon my attempt to hijack your thread, but does anybody know of a solid solution in this respect? Or even a good way to just simply sync and manage larger datasets or files over www.

I am not so concerned about version tracking/management so much as just a good way to sync large amounts of data while having a git–like CLi arsenal. Or a comparable CLi solution, or API with decent version tracking and solid support.

I have mounted google drive as a linux volume which is buggy to say the least. I have used github, and local git repos, but I am wondering if I am missing out on anything.

For example, BitTorrent has a Sync tool that seems close to what I want.

If you need source control and large binary files then use Perforce. It's pretty solved.

Thank you!

I don't have an answer, but I have an additional complaint. Git itself is pretty terrible at dealing with large repositories if for no other reason than because "git clone" has no resume feature. If your internet connection is interrupted at any point during the initial clone it dies and you have to start over. This means that on an unreliable internet connection it is nearly impossible to clone large repositories.

A git clone is not the same as just rsyncing a .git directory, but if you know Git, the latter can be made to work. You may want to GC the source repo first, and you'll need to manually add a remote and possibly clean up branches after you're done but a quick bash script can finish that up a treat too. (Good task to practice bash scripting on, if you're not familiar with it.)

I've done that, waste of time. There's just no excuse for git being unable to resume a clone.

I've never heard this complaint before. I would have never guessed it would be a problem. Anyway, for the clone, you could just use a different command (such as rsync) to download the other Git repository to your HD, and use git clone on the local path. There is little difference since git clone grabs everything.

Subsequent pulls can be done directly to the remote repository.

Most game design companies use Perforce specifically for this reason, they have massive asset files that also need to be version controlled and Perforce usually does this better than other choices such as git/hg/svn. Sadly Perforce is proprietary and quite expensive, but it's basically the gold standard in game dev.

I haven't used it, but https://git-annex.branchable.com/ sounds like it fits your bill rather nicely.

I'm currently working on a geographically distributed digital archives project. We have hundreds of collections of large binary files (video, audio, archival-quality scans) and their associated textual metadata. We're using Git and git-annex to manage all these files and sync them between locations.

Some of the collection repositories are pretty big. One collection has 21000 5MB JPGs. Another has 283 190MB files. Some operations do indeed take time, but you expect that when you're working with so many files. git-annex is awesome!

Late response here, but I will have to try this one out as well. git-annex was [virtually] the only result returned from my breif google search. Weird that P4 was no where to be found. Going to have to try both of them out once I get google fiber. :)

i've been using unison to sync a checked-out source repository between a host linux machine and a windows vm, over ssh.

if you only want uni-directional synchronisation, then maybe just use rsync?

this is distinct from your use case as i'm not using it to sync with remote machines, but it might be worth checking out.

http://www.cis.upenn.edu/~bcpierce/unison/

Thanks shoo. I wish this thread wasn't dead so that maybe someone would respond that has used unison. Will check it out if I don't end up loving P4 or git-annex :)

some of these problems / symptoms can be framed as a dependency-management problem, which can be addressed by other tools: http://blogs.atlassian.com/2014/04/git-project-dependencies/

that said, it's not like that doesn't introduce other problems

I've used submodules before to separate game assets and it just ended up being more trouble than it's worth. The problem is that it separates the history between your two repositories so it takes more time to figure out which asset commit relates to which code commit. In the end, we just merged everything back into one repository.

Agreed. I have a but of a hack that I've stayed to use - I've been using SVN for my game assets and I have git hooks to pull the correct version. Still not optimal, but better.

I'll throw in a word for git-fat[1]. It works great and has a dead simple implementation that actually has a real copy of the file in the working copy of the repository (as a posed to symlinks).

https://github.com/cyaninc/git-fat

Would Android's `repo` be a good solution to this?

Yes and No.

Repo was born to resolve this kind of problem, but it might not be a "good" one. We (git/gerrit team at Google) are working on bring cross-repository atomic submit and other stuff to git/gerrit and our goal is to replace repo with git submodule.

Here're a very brief slides[1] and notes[2] about this topic at this year's Gerrit User Summit.

[1] https://docs.google.com/presentation/d/1qG1eAiDmyozZBiVE6R4A...

[2] https://docs.google.com/document/d/1a2eFhVr1HUiKOjhaRHn_89mf...

That's very interesting, as someone who occasionally contributes to the odd Android fork. I kind of feel like repo is this weird semi-black box which does things I don't expect, like reverting my topic branches back to the remote branch but leaving the name in place (it does this if your topic branch does not have a remote tracking branch, and you run repo sync). I feel that at a minimum repo should play nice with standard git workflows. I should probably just read the repo source but I can't imagine why this behavior would be a good idea, and I'm usually left feeling like I would rather just use git directly than repo most of the time.

lol @ handling big repositories

Reply on news.ycombinator.com