Hacker News Reader

Comment by thewopr | Hacker News Reader

Here's why I'm excited about this: Data Versioning

Github is awesome for science. But my code and workflows often are just as reliant on the data I have as the code I've written. With Github, I always have to treat my data separately. A different workflow, a different storage location, different (and often manual) versioning.

Now, I can start integrating my data into the same workflow. No size limits. Just drop it in and version it like everything else. 1GB? No problem. 500GB? Just pay the money.

This is especially awesome because, as a scientist/dev, I do not want to stand up my own infrastructure and servers/vms. I don't want to do linux updates. I don't want to worry about things going down. It just needs to be there when I need it. When I don't need it anymore, I'll take it down. Done and done.

View on HN

We've build a Git extension (git-bigstore) that helps manage large files in Git. It uses a combination of smudge/clean filters and git-notes to store upload/download history, and integrates nicely with S3, Google Cloud Storage, and Rackspace Cloud. Just plop in which filetypes you'd like to not store in your repo in .gitattributes and you're good to go. Our team has been using it for a while now to keep track of large image assets for our web development projects. Cheers!

https://github.com/lionheart/git-bigstore

If you're looking for an open source system for managing fat assets in git, we made a tool that integrates with S3: https://github.com/dailymuse/git-fit

How does this compare with git-fat?

Git-fat uses smudge/clean filters. We used git-media which employs the same technique, and it didn't work well for us. See the section about git-media in the readme.

I maintain a similar tool to manage large files in Git [1], but went the clean/smudge filter route. I think git-media gets into states where it can process files twice even though it shouldn't. From the Git docs:

> For best results, clean should not alter its output further if it is run twice ("clean→clean" should be equivalent to "clean"), and multiple smudge commands should not alter clean's output ("smudge→smudge→clean" should be equivalent to "clean").

So this isn't the fault of clean/smudge filters, just the way they were used with git-media.

[1]: https://github.com/lionheart/git-bigstore

We experimented with smudge/clean filters with our own implementation and they just didn't seem like the right solution for fat asset management.

The most frustrating problem was that filters are executed pretty frequently throughout git workflows, e.g. on `git diff`, even though assets rarely ever change. The added time (though individually small) created a jarring experience.

I'd also be curious how git-bigstore addresses conflicts. It seems like a lot of the filter-based tools out there don't handle them well for some reason.

I like the look of this. Thanks. Will investigate

Have you ever looked into using an s3 backed git repo via jgit? Not sure how many files/revisions you have but it may be a cost effective alternative that you can start using today.

We use it for auto deploys from autoscaled instances cuz github has poor uptime.

No, but I don't understand what this is. Java git? Bound to eclipse? I don't really understand how this would be different than me storing a .git repo locally and replicating it to S3.

jgit is a java implementation of git, that just so happens to have built in s3 support.

You can get a self contained cli (jgit.sh) from here: https://eclipse.org/jgit/download/ . It does not have an eclipse dependency.

you can then just add an s3 remote and push to this remote as part of your CI flow.

Here is a decent article that describes the end-to-end: http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...

I love my jgit s3 git repos. It is super cheap almost free git storage/backup.

I would ramp down your excitement, this doesn't solve any of the problems that Git has around large files. Dropping in a 1GB data file into your Git repo will cause problems for you, even if the server doesn't block it like GitHub does.

They mention large binaries on the product page enough times to make me think that they're rewriting Git.

I'm not sure that a server-side rewrite addresses the core issue. Git is designed so that a git fetch pulls the entire repo down. The reason large binary files are a problem is because you are going to pull down the entire history of those files in order to clone the repo. Unless they're also shipping a custom git client (unlikely) then I don't see how they can solve this problem on their end.

edit: Come to think of it, the feature they could potentially implement their side is somehow making it so binary files have their history purged somehow. I'm not a git expert but if they are able to trick git into think the files are new, and have no history, each time they change then I guess that could work.

the core issue is not fetches. The last time I tried to use git to manage scientific data, commits sometimes took ~10 hours. And this was with all text files, not binaries. This was ~100k files and 20 gigs of data.

Unless that version of Git is also running on your computer, it doesn't change anything. The problem is with local files, not remote ones

Oh sure, but at least my limitation in the future will be Git being able to process and handle large files (and maybe the cost of storage) and not Github's file and repo limit.

It's reasonable to assume that Github's limits are caused by Git's limits. When Git begins supporting large files, most likely Github will be able to raise their limits as well.

You sure about that? Github is a SCM company, not a bulk storage company. When your business is not storage, arbitrarily large storage requests aren't interesting...

Github has gone through great lengths to begin supporting diffs on all sorts of files, 3d volumes, images, and all sorts of assets. I'd be willing to bet that some sort of bundled asset management is someone's dream project at Github.

I've run into problems with large-ish files in git repos, binaries accidentally committed etc. I'm genuinely curious, is there a good way to use git for the size repositories that you mention?

Git is great for smaller binaries. Ideal in fact, given that it stores differences between revisions as binary deltas. For large >1GB files, I believe the diffing algorithm is the limiting factor (I would be interested in getting confirmation of that, though). For those files something like git-annex is useful (http://git-annex.branchable.com/)

I've used git to push around a lot of binary application packages and it's very nice. Previously I was copying around 250-300MB of binaries for every deployment--after switching to a git workflow (via Elita) the binary changesets were typically around 12MB or so.

Hmm, I'll have to take a look at git-annex, I've run across it but never investigated. Thanks!

I don't know of a great solution for Git, but I've heard that Perforce is more suited to handling large binaries - I believe it's used in many game development studios, where binary assets can number in the high gigabytes.

Game developers need both support for large files and for exclusive locking, since assets like images, audio and 3d models are almost never mergeable.

I've had no trouble with non-github git repos handling text files well into the hundreds of megabytes. I haven't pushed the limit on this yet, but for me, personally, my datasets are often many hundreds to thousands of individual files that are medium sized. So it tends to work fairly well. Singular large files may not scale well with git.

Domino has a nice solution for version control and reproducibility in the context of analytical / data science work: they keep a snapshot of your code, data, and results every time you run your code. So it’s like version control plus continuous integration. Supports large data files, R, Python, etc. http://www.dominodatalab.com/

Yeah, but their system is super simplistic and doesn't support half of the diverse and robust operations that Git and Github do. Can I see version history? Are changes compressed into diffs for text files? etc, etc.

Git stores whole objects, not diffs, in case you were confused.

Git stores diffs between revisions too: see https://www.kernel.org/pub/software/scm/git/docs/git-pack-ob...

Packs happen automatically sometimes per https://www.kernel.org/pub/software/scm/git/docs/git-gc.html

You may not be aware that data versioning is a built in feature of S3[1]. No VMs, no management, just put all stuff in one place and it's versioned automatically.

As others have pointed out Git is not a great way to do big data versioning. Git is almost exactly the wrong tool for the job, almost any other revision control system handles large file versioning better because you're not expected to clone entire archive histories to everyone.

[1]: https://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket...

I came to say the same thing. We have a lot of videos in one of our projects and therefore we can't host it on github due to the size limits.

.gitignore isn't a viable option?

Not if you need them versioned.

> I don't want to do linux updates.

I'm no EC2 expert, but I believe you still have to do linux updates if you are running on AWS.

If you use Amazon's AMIs they do a lot of updates for you. Recently, they updated their AMI for the bash and SSL security exploits. If you have a permanent instance, you would have to run some updates yourself, but if you use auto-scaling, you can just update the group and then when the instances cycle, the new updates will be applied.

This is also the case with Ubuntu (the most popular Linux distro). The AMIs get updated regularly.

Not for their hosted version management solution, I assume.

What are the disadvantages to using plain Git for your data?

Git's storage model is such that every commit is physically a snapshot of the entire work tree. Git makes this efficient with delta compression (essentially, deduplication) which is extremely efficient for text (in fact an entire git repository can be smaller than a single SVN checkout) but is less effective with large binary files since changes don't produce compressible deltas.

> in fact an entire git repository can be smaller than a single SVN checkout

Only because svn downloads the plain text of the server's version of every single file from the revision, so that "svn diff" doesn't have to hit the server.

Reply on news.ycombinator.com