I love everything about how prompt manipulation is turning out to be a major weakness of exposing LLMs to users.
It feels like this vulnerability reflects how LLMs are indeed a huge step not just towards machine intelligence but also towards AI which behaves similarly to people. After all, isn't prompt manipulation pretty similar to social engineering or a similar human-to-human exploit?