Similar to you I keep mine in a git repository, but I use dotbot[1] to manage the symlinks to the proper paths.

[1] https://github.com/anishathalye/dotbot