It is possible to require confirmation before `ssh-add` allows to use a key. From `man ssh-add`:

     -c      Indicates that added identities should be subject to confirmation
             before being used for authentication.  Confirmation is performed
             by ssh-askpass(1).  Successful confirmation is signaled by a zero
             exit status from ssh-askpass(1), rather than text entered into
             the requester.