Isn't this what git lfs[0] sets out to solve?

[0] https://git-lfs.github.com/