Hacker News Reader

Too bad git is not good for large files - http://kerneltrap.org/mailarchive/git/2006/2/8/200591

Well, the extent to which this matters depends on how often you run it. Even if the hashing for a 10gb database dump takes ten minutes* , it will still get backed up ten minutes after the hour, every hour. There are very likely better solutions, but it would be easy to set up, and better than nothing.

(Besides, any solution that has to deal with a complete db dump is going to have to hash, diff, or otherwise process a large volume of data.)

* As a single data point, sha1 just ran for me at approx. 800mb/min.

a better fix is to mysqldump every table individually, and then git commit those as distinct .sql files. This way less-often updated data (reference tables, etc) tend not to need updating, and your hash generation is typically smaller. Plus, you can skip log/session type tables if you are that way inclined to use them. :)

That is exactly what 1 client of mine does for Postgres.

Looks like the same emails I was pulling up after switching from SVN and discovering my gigabyte-plus census files were exhausting Git's memory. Sadly, in the game industry many of us are stuck with Perforce just because it handles gigantic files (like a DVD image), has a visual client and is marginally better than Microsoft's offering.

Reply on news.ycombinator.com