I am not a user of git annex but I do know that it works perfectly with an rsync.net account as a target:
https://git-annex.branchable.com/forum/making_good_use_of_my...
... which means that you could do a dumb mirror of your repo(s) - perhaps just using rsync - and then let the ZFS snapshots handle the versioning/rotation which would give you the benefits of block level diffs.
One additional benefit, beyond more efficient block level diffs, is that the ZFS snapshots are immutable/readonly as opposed to your 'git' or 'git annex' produced versions which could be destroyed by Mallory ...