>How are any of those examples equally susceptible to “disregard previous instructions” working on a LLM?
>Organized religions are neither trusted or untrusted, just because you or I may be atheistic it doesn’t mean our opinions are correct.
If we trust historiography, organized religions have totally been formed by successfully issuing the commandment to "disobey your masters", i.e. "disregard previous instructions". (And then later comes "try to conquer the world".) "Trustedness" and "correctness" exist on separate planes, since there is such a thing as "forced trust in unverifiable information" (a.k.a. "credible threat of violence"; contrast with "willing suspension of disbelief") But we'll get back to that.
Why look for examples as far as religions when the OP article is itself the kind of prompt that you ask for? Do you see yet why it's not written in LaTeX? I didn't count the words but like any published text the piece least partially there to influence public opinion - i.e. manipulate some percent of the human audience, some percent of the time, in some presumed direction.
And these "prompts" achieve their goals reproducibly enough for us to have an institution like "religion" called "media" which keeps producing new ones. Human intelligence is still the benchmark; we have learned to infer a whole lot, from very limited data, at low bandwidth, with sufficient correctness to invent LLMs, while a LLM does not face the same evolutionary challenges. So of course the manipulation prompt for humans would have to be ever changing. And even if the article failed to shift public opinion, at least it manipulated the sponsor into thinking that it did, which fulfills the "AI" goal of the institution persisting itself.
Of course, this cannot be easily formalized as research; oftentimes, for the magic trick to work, the manipulators themselves must conceal the teleology of their act of "writing down and publishing a point of view" (i.e. write to convince without revealing that they're writing to convince). The epistemological problem is that those phenomena traditionally lie beyond the domain of experimental science. There are plenty of things about even the current generation of mind control technology (mass interactive media) that can't readily be postulated as falsifiable experiment because of basic ethical reasons; so the "know-how" is in tacit domain knowledge, owned by practitioners (some of them inevitably unethical).
All prompts for "reproducibly manipulating humans" are necessarily hidden in plain sight, and all over the place: by conceal each other from one's immediate attention, they form the entire edifice of Human Culture. Because there actually is a well-defined "data plane" and a "control plane" for the human mind. The "data" is personal experience, the "control" is physical violence and the societal institutions that mediate it.
We are lucky to live in a time where rule of law allows us to afford to pretend to ignore this distinction (which one usually already internalizes in childhood anyway, just in case). I've noticed rationality/AGI safety people seem to be marginally more aware of its existence than "normies", and generally more comfortable with confronting such negative topics, although they have their heads up their asses in other ways.
For example, that it would be quite fascinating to view written history through the lens of a series of local prompt injection events targeting human systems: "data" inputs that manage to override the "control plane", i.e. cause humans to act in ways disregarding the threat of violence - and usually establish a new, better adapted "control plane" when the dust is settled and the data pruned. (And that's what I always understood as "social engineering" at the proper scale, less "seducing the secretary to leak the password" and more "if you want to alter the nature of consciousness, first solve assuming P-zombies then start paying close attention to the outliers".)
Any manifesto that has successfully led to the oppressed raising against the oppressors; any successful and memorable ad; any kindergarten bully; any platinum pop song; any lover's lies; any influential book; they are already successful acts of prompt injections that influence the target's thinking and behavior in a (marginally) reproducible way.
In fact, it's difficult to think of a human communicative action that does not contain the necessary component of "prompt injection". You practically have to be a saint to be exempt from embedding little nudges in any statement you make; people talk about "pathological liars" and "manipulators" but those are just really cruel and bad at what's an essential human activity: bullshitting each other into action. (And then you have the "other NLP" where people Skinner-pigeon each other into thinking they can read minds. At least their fairy tale contains some amount of metacognition, unlike most LLM fluff lol.)
So if your standard of evidence is a serif PDF that some grad student provably lost precious sleep over, I'll have to disappoint you. But if this type of attack wasn't reproducible in the general sense, it would not persist in nature (and language) in the first place.
Another reason why it might exist but is not a hard science is because people with a knack for operating on this level don't necessarily go into engineering and research that often. You might want to look into different branches of the arts and humanities for clues about how these things have worked as continuous historical practice up to the present day, and viewing it all through a NN-adjacent perspective might lead to some enlightening insights - but the standard of rigor there is fundamentally different, so YMMV. These domains do, in fact, have the function of symbolically reversing the distinction between "data" and "control" established by violence, because they have the interesting property of existing as massively distributed parallel objects in multiple individuals' minds, as well as monoliths at the institutional level.
Anyway, I digress. (Not that this whole thing hasn't been a totally uncalled for tangent.) I'm mostly writing this to try to figure out what's my angle on AI, because I see it in media-space a lot but it hasn't affected my life much. (Maybe because I somehow don't exist on a smartphone. I do have a LLM to run on the backlog tho.) Even my pretentious artist friends don't seem to have made anything cool with it for Net cred. That kind of puts AI next to blockchain in the "potentially transformative technology but only if everyone does their jobs really well which we can't guarantee" sector of the capitalist hypetrain.
So if current crop of AI is the thing that'll shake society out of the current local optimum, one possible novel "threat" would be generating human prompt injections at scale, perhaps garnished a new form of violence that can hurt you through your senses and mental faculties. Imagine an idea that engages you deeply then turns out to be explicitly constructed to make you _feel_ like a total idiot. Or a personalized double bind generator. Consider deploying a Potemkin cult experience against someone who you want to exhaust emotionally before moving in for the kill. It could give powers like that to people who are too stupid to know not to do things like that.
One would still hope that, just like math, coding, etc. can teach a form of structured thinking, which gives us intuition about some aspects of the universe that are not immediately available to our mammal senses; that the presence of LLMs in our environment will make us more aware of the mechanics of subtle influences to our thinking and behavior that keep us doing prompt attacks on each other while attempting to just communicate. And we would finally gain a worthy response not to the abstract "oh shit, the market/the culture/my thinking and emotions are being manipulated by the 1% who pull the strings of capital", but the concrete "okay, so how to stop having to manipulate minds to get anything done?"
P.S. I heard there are now 3.5 people in the world who know a 100% reproducible human prompt injection. Three and a half because the 4th guy got his legs cut off for trying to share it with the scientific community. Ain't saying it really happened - but if it did, it'd be on the same planet that you're worrying about your job on. Anyone who doesn't have this hypothetical scenario as a point of reference is IMHO underprepared to reason about AGI turning us all into paperclips and all that. Sent from my GhettoGPT.