I'm thinking that rather than using git for versioning the binary artifacts as well, you tag and version your git repo, then tag and name/label your artifacts in another storage service. You then allow a build tool to assemble from both locations.