I just track ~/ directly and add appropriate .gitignore entries to those files which are ephemeral or not part of my local configuration:

https://github.com/uggedal/dotfiles/blob/master/.gitignore