git-annex is an ugly hack to, basically, reimplement git-with-this-feature on top of git. What I'm suggesting would be to reimplement git itself on top of distributed object storage -- effectively, to split git into two tools:
1. one that commits treeish diffs into objects, and then stuffs them into arbitrary content-addressable storage;
2. and one that manipulates (local or remote) mutable tables of refs to content-hash-URNs -- or "git repositories" for short.
The key component would be a facility (either a library or an OS service) for resolving content-hash-URNs into file-descriptors in some content-addressable storage system. The critical point is that this would not be a git component -- the committing tool wouldn't be in charge of where its objects go, and the repository tool wouldn't be in charge of resolving their URNs. Instead, both would just be relying on the host to decide how to get objects from/put objects into storage, and it's the host that would have a distribution strategy set up for all of its content-addressable objects, not just the ones that happen to be attached to git.