Yeah, git with binary files is not fun. Have you tried git annex?