I sort of disagree that LLMs don’t have the same pitfalls. LLMs aren’t recording everything they are trained with, like humans, the training data affects a general behavioural model. When answering, they aren’t looking up information.

As for being “primed”, I think the difference between training, fine tuning, and prompting, is the closest equivalent. They may have been trained with anti-scam information, but they probably haven’t been fine tuned to deal with scams, and then haven’t been prompted to look out for them. A human who isn’t expecting a scam in a given conversation is much less likely to notice it than one who is asked to find the scam.

Lastly, scams often work by essentially pattern matching behaviour to things we want to do. Like taking advantage of peoples willingness to help. I suspect LLMs would be far more susceptible to this sort of thing because you only have to effectively pattern match one thing: language. If the language of the scam triggers the same “thought” patterns as the language of a legitimate conversation, then it’ll work.

To avoid all of this I think will require explicit instruction in fine tuning or prompts, but so does everything, and if we train for everything then we’re back to square one with relative priorities.

The problem is that the attacker can try a gazillion times and only needs to succeed once.

This is where it is different from the human case, where the human will get bored after 3 phishing attempts and closes their email program.

Everyone has their scam threshold. Even the most well-educated people can be pwnd if caught when distracted, or tired, or the phish looks legit by coincidence[0]. Or you just keep cranking up the urgency and stakes involved.

Possibly related: confidence schemes and magic tricks. As the adage goes, one of the best way to make a magic trick work is to make it much more elaborate, and/or invest much more in its setup or execution, than any reasonable person would ever expect.

--

[0] - A fake package delivery mail that, by chance, came at the exact time you expected one for a real order, and with very similar details. Or fake corporate OneDrive deletion e-mail that came just after your system was migrated in a process that could involve deletion of old OneDrive files.