It may be sacrelege but you could store your binaries in hg and in hg use a git submodule for the git repos that were used to build those binaries.

This is what I'm doing after a disastrous excursion into git-fat.

But really, if git's maintainers don't want to store binaries, why shoehorn it in? Use the right tool for the job.