If you forward SSH agent, anyone with privileges to your sockets can use the agent. So for example when you log into some production machine, a process waiting on that machine connects to your agent and has access to... everything. (If you're agent caches key password you won't even get notified)
Assuming your local agent silently signs all requests. This attack doesn't work if e.g. your Yubikey is configured to require a physical tap for every authentication.
Using ssh-add -c when adding keys means it asks for user confirmation (but can still cache the passphrase) each time key use is requested. This can mitigate some of the risk here in the absence of other solutions.