The SSH agent isn't divulging the keys. The OP is decrypting the keys from the underlying storage. The API used to protect the keys allows the keys to be decrypted without explicit password entry. The OP has the necessary password to decrypt the keys, so the API allows it. The SSH agent isn't involved.