Having nested git repositories is a solved problem both in git and in git-annex: use submodules.

https://git-annex.branchable.com/submodules/