From a security standpoint, what you want to do is less important than what you can do, and the access controls you use currently -- public or public + private -- just aren't fine grained enough.
We agree. We spent a lot of time debating this.
We originally set out to build a gem, but realized it was so much easier to use through GitHub. Unfortunately, we can't provide more granular access due to what the API gives us.
We plan on eventually releasing said gem, as noted in the "I don't use github" link on the front page :).
On deploy posting the Gemfile is something that came to my mind too and would work.
You could probably hack around it by creating an integration account that only has access to the repositories that you want to integrate. If you're asking for only access to the Gemfile and not the repository, that's far outside the security model of GitHub, and even Git.
Maybe a good alternative for people security conscious would be a manual Gemfile upload?