Is ssh-agent even worth the trouble?
You can create a new key _on that machine_ and authorize _that key_ for the specific purpose you need.
Or you can also just clone the repo on your local machine and add the remote machine as a git repo remote. Then you proxy all commits between github and the remote machine.
If you don't want to proxy commits between three repos (github, your local, your remote) then you could install sshfs on your local machine and sshfs-mount your remote machine.
Regardless; if you're running git on the remote machine then the root user of that remote machine would have access to the key. Why let them have access to _your_ key when they're only administering a machine for that repository? Let them have access to a shared key.