Personally I have a confirmation dialogue for any agent forwarding request. When I require it, this seems like the best compromise. (I still have my key decrypted in memory, I just get a "Use key?" window that I need to hit enter for).