GitLab Annex: Large Binaries with Git
about.gitlab.com
about.gitlab.com
https://git-annex.branchable.com/
Edit: Just found the relevant quote from the article
> In GitLab 7.8 Enterprise Edition this problem is solved by integrating the awesome git-annex.
I guess the repeated branding of "GitLab Annex" just seems a bit strange to me.
Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files.
Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.
It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar).
Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.
For example, imaging you work at a video production company where you have videos on each page of your website. (Let's also imagine that you're not using vimeo or youtube to host them.) You might make a branch of the home page with a different video, and you want to coordinate changes across pages.
git-annex is amazing for this use case, as you don't have to actually have a local copy of videos that you're no longer using.
I'm not even sure it's possible for a rebuild to produce exactly the same binary down to the byte (internal versions, timestamps, etc). And even if it could, it would require guaranteeing the build system hadn't been changed, patched, etc.
I can only assume people that don't commit their binaries don't have to look at crash dumps very often...?
We're quite excited about GitLab Annex and curious to hear what people think about it.
We have to suffer under Perforce for the first of those reasons (we have artwork and CAD files none of which are text). Sadly, P4 is pretty much the only game in town for a medium sized organization with binaries (someone large like Google can afford to replace it with an in house solution but that isn't worth it for most of us).
I tried annex a few years ago and it was also painful, but perhaps it's gotten better. Certainly I miss the power of git when using P4.
We'd love to get feedback on it.
PlasticSCM has a hybrid approach that is worth studying.
Taking the game studio example, why would two developers/artists/whatever need to work on the same asset at the same time? Locking stops them from stepping on each other's toes, but it also means one has to wait for the other to finish their task before they can do anything.
Seriously, with the tens of thousands of assets in a medium to large game, the way they can be reused across the game, and the peculiarities of art and in-house tools, it's not uncommon for two artists to try to modify the same asset. Sometimes it's accidental, sometimes it's necessary, sometimes it is an oversight, sometimes it is an organisational issue as you say... but locking detects the conflict and prevents it from turning into wasted hours or days.
Relying on a separate tool to manage this issue is a notable increase in friction. Since the SCM handles changes and conflicts for source code (and thankfully allows merging), why wouldn't it do so for art?
True to some extent. (Large files I will agree with.) That wasn't his only argument: the other was about large repos. I've seen many places try to attempt a 1:1 conversion of a Perforce depot to a git repository, and this will likely not work well: the Perforce depots are simply too large. A Perforce depot typically, in my experience, contains the entire source code for the entire company. A Git repository would be suitable for a subtree in that depot that represents a logical project, often managed by a team of related co-workers. So one Perforce depot would correspond to multiple git repos; however, like I said, I feel like people try to shoehorn the entire depot into a single repo.
The downside is you have multiple repos, whereas before you had a single large repo, and any cross git-repo changes would be made in a single changelist. In git, you'll need multiple commits, and some way to manage inter-repo dependencies in those changes. (Such as semantic versioning, and a decent build system.)
The downside of Perforce that I ran into while working with it is that it lacks git's branching model, many of git's day-to-day conveniences (I so miss git add -p, git commit for just pushing out small fixes that either don't need review or for which review is trivial), and frankly, the fact that a commit hash represents a set state. (Between creation of a changelist, running tests, and submitting the changelist, someone can break you; this is not possible in git, as the push will fail.)
In the long run, I greatly prefer the extra management work of git for the power it brings with its branching model.
> I don't think he was that negative.
> > Most grownup software shops use Perforce
In my reading of this, there is an insinuation that shops using git are not "grownup"; I would call this disrespectful, myself.