AddKeysToAgent defaults to no. Ptrace might also be disabled, depending on system. I would be more concerned about keyloggers, or any tricks that result in me running something else than the real ssh client (e.g. custom program somewhere in PATH).