I feel like this missed the relatively straightforward https://github.com/emacsorphanage/git-gutter package which I feel does a pretty good job and does exactly what I want.