Hacker News Reader

Comment by AndrewDucker | Hacker News Reader

SVN works better when you're dealing with large binary files.

If you're bundling in all of your dependencies, or some images/models/sound files then Git gets very big, very quickly.

Yes, and I am surprised that this didn't up as answer #1 in response to the original question. This problem is an absolute showstopper if you need to manage binary assets (which for something like video games, you absolutely do -- you want to know that your data is the right data for the current version of the code, because the data is changing all the time too).

In The Witness (http://the-witness.net/news) we have 20GB of data checked into svn. Try that with git.

You have to use something like git-annex. Possibly some other options that others might know of.

Using git-annex would probably fix a lot of these problems.

On the other hand, Perforce is the game industry standard. As much as engineers complain how terrible P4 is in comparison to distributed options, the other teams working on the title (artists, designers, QA/Test) save a huge amount of time by simply not fucking things up for everyone else.

When there's 250 people working on a title, producers will call to lock the entire depot down so that Intern Joe Blow from Test/QA can't check in a broken build and waste the time of everyone who is working on the project.

I've administered depots (along with overseas proxies) with a 1TB head revision. It was amazingly fast despite how huge the data set was. The only exception was syncing a new workspace, which is easy enough to work around with weekly snapshots and rsync.

Try that with git.

My last company did just that. The repository wound up being 30gb at the end of the project (which was a 3d facebook game). We tried to host on a company hosted github, but got kicked off because it couldn't deal with a repository that size[1]. Luckily, we only had one person try to check out the repository at the end of the project. It took them 14 hours for the initial check out.

[1] It was never made clear to me whether it was our hardware that had the problem or the github self hosted software. Either way, it was unreasonably slow.

Use git-annex. I have 100gb of video footage using git annex.

This just happened at work like 2 weeks ago. Someone checked in a large (4g) binary file into SVN repo and it crashed the whole system. Everyone at the company was blocked for half a day before they got it fixed (temporarily to unblock everyone). But it took another 2 days and a lot of heated debates between the release manager and devops before everything went back to normal. There were talks about moving to git before the incident, but this really pushed management to finally pull the trigger. 2 lessons here: 1. SVN is bad at large binary files; 2) centralized source control's one way of doing things also gives you one way of disaster recovery.

I'm not surprised it choked on a 4Gig binary file.

But when we have dozens of multi-MB binary files, which change every week or so, then after a year we're ending up with a massive repository to pull down.

`git gc` packs files with delta compression. I have a rather big git repository (daily mirroring a 25 gig repository with some large dvd iso images). And it seems to work quite well.

How does mercurial do in that situation? I know it uses a binary diff algorithm under the hood, but are there other showstoppers for storing giant files in mercurial?

They don't store well as compressed binary diffs, and they can cause high memory consumption during various operations. If you absolutely must store large binary files then there's a large files extension.

The largefiles extention is now bundled with mercurial (since 2.0):

http://mercurial.selenic.com/wiki/LargefilesExtension

[edit: this isn't quite a solution, more of a work around -- but one that can be expected to be installed by default.]

Also, a memory hungry monster, while it's forwarding them.

Reply on news.ycombinator.com