Only developers that have sent me their public ssh key have access to

Same way you would for any other server. And I mean that 100% literally, given that at the command-line level the remote is simply a URL: https://git-scm.com/docs/git-remote