I guess this is more an issue with GitHub OAuth permissions which are not fine grained enough. What you want is read-access to all repositories.

I would not give anyone access to all my private repositories, however.