You can restrict keys when added so that they are only available on specific hosts: https://www.openssh.com/agent-restrict.html