However, in reality the difference is probably marginal: The majority of the (ex-)employees has no bad intent and couldn't care less about old company stuff once they leave or change roles. Those who plan to do something bad, will also keep copies of passwords from an online system.
To be safe you must rotate once someone leaves or trust your firewall.
pass supports different access in different parts of the repo, just use different .gpg-id files. But the whole system is kind of geeky. If you employ others than hackers familiar with the command line, gpg and git, it's not really an option.