ssh-add has a -c option, which uses ssh-askpass for confirmation. Unfortunately it doesn't give much information, but you hope you would notice it coming up when you didn't initiate it. You may also be able to do something with gpg-agent, it seems to have more options.
Oooooooh, I missed that one. It would have been better if it wasn't an option, but default behaviour, but I guess we can't win every time...