But they can tell the agent to authenticate with any key loaded in your agent, not just the one you used to ssh into the machine you forwarded your agent to
So e.g if you have a distinct ssh key for GitHub and a different one for all other uses and you ssh to a compromised server with agent forwarding, the attacker can then ssh to GitHub as you.
() there was a vulnerability not too long ago involving getting the remote agent to load arbitrary shared objects for remote code execution, which obviously changes things
I would not be surprised if there is some way to attack this into getting a private key, but it would either be a direct attack on the agent code like sending it malformed messages to somehow get remote code execution to then read the key. Or some more complicated attack on the cryptography where you repeatedly force it to auth and can somehow use the results to reduce the key space needed to brute force the key - along the lines of a known plaintext attack.
But I'm also just a hobbiest here who has looked a little into the security model but I am by no means a cryptography expert, so take all of this with a grain of salt.
There are alot if details in this that can go wrong. I seldom use agent forward in unknown/undesigned environments because of this.