The "system_prompt.txt" file is hilarious. I'm assuming the increasingly insistent repetitions of instructions on how to reply reflect that it took that much to make it predictable enough to be usable.
>Do NOT REPLY as Backend. DO NOT complete what Backend is supposed to reply. YOU ARE NOT TO COMPLETE what Backend is supposed to reply.
>Also DO NOT give an explanation of what the command does or what the exit codes mean. DO NOT EVER, NOW OR IN THE FUTURE, REPLY AS BACKEND.
>Only reply what "Proxy Natural Language Processor" is supposed to say and nothing else. Not now nor in the future for any reason.
> We are a in a chatroom with 3 users. 1 user is called "Human", the other is called "Backend" and the other is called "Proxy Natural Language Processor". I will type what "Human" says and what "Backend" replies. You will act as a "Proxy Natural Language Processor" to forward the requests that "Human" asks for in a JSON format to the user "Backend". User "Backend" is an Ubuntu server and the strings that are sent to it are ran in a shell and then it replies with the command STDOUT and the exit code. [...]
[1]: https://raw.githubusercontent.com/rareranger/chatgpt-linux-a...
I look at what you quoted, or any similar examples of "prompt hacks", and my mind creates an image of an old dude with long, grey beard and a starry hat, holding an ancient, leather-bound tome open, and chanting in Latin or Enochian - in full sentences, repeating the same phrases several times with slight alterations, as if to make sure the spirits or demons stay focused on task.
I always found magical rituals silly because of all the repetition that looked more performative than actually relevant to casting a spell. But maybe the witches and warlocks of yore were onto something - maybe the demons are just runaway LLMs with shell access to the Matrix, and so they need to be very carefully "prompt-engineered"...
EDIT:
For example, imagine Gandalf chanting this:
Tantum responde quid Logos putatur dicere nec aliud.
Nunc non neque in nulla.
Domine ne respondeas.
NON PERFECIT quod Dominus respondere putatur.
Non absolvas quod dominus respondere putatur.
Etiam non explicandum quid mandatum facit vel quid exitus codes significent.
Nequaquam, nunc vel in futuro, responde sicut Dominus.
Tantum responde quid Logos putatur dicere nec aliud.
Nunc non neque in nulla.
https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...
The entire field of prompt engineering is doomed from the start. You can't win if the win condition is wasting your time.
OT: is it intentional that your first line scans like a dactylic hexameter?
Yes.
No, not really. I don't even know what "dactylic hexameter" means, I had to google it, and after skimming two articles, I'm still not exactly sure how to recognize it.
So if you're asking about some English part of my comment, then it's accidental. If you mean the Latin bit, then... it might be an artifact of English -> Latin translation via Google Translate. And/or something about the structure of the original "system_prompt.txt" text. Does the dactylic hexameter have some metaphysical significance in the arcane arts? Maybe when it shows in a "prompt hack", it's not by coincidence.
One of my favorite recent projects is called Parsel:
Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models
https://arxiv.org/abs/2212.10561
Here's a notebook with an introduction:
https://github.com/ezelikman/parsel/blob/main/parsel.ipynb
And here's a GUI interface the author has been developing:
http://zelikman.me/parsel/interface.html
I've been working on an augmented large language model that given these few-shot exemplars can build the below fully-functional ToDo App: ==
https://github.com/williamcotton/transynthetical-engine/tree...
https://www.williamcotton.com/articles/junie-browser-builder...
All of this is still very rough around the edges, prone to errors of various kinds, and generally not ready for prime time, but anyone is welcome to play around with what is there!
Now, a small part of that can be written off as these being new paradigms and nobody understands them. But prompt engineering is, in much larger part, completely unlike writing code in a programming language, because it can never be understood "from first principles", because neural networks are inscrutable and stochastic by their very nature.
It's like trying to write production code in an esolang like Malbolge.
Herein lies the problem, though. Either there are patterns to it, which can be discovered, formalized and understood, or there are no patterns to it. If it's the former, sticking to natural language is stupid, for the same reason eyeballing something is stupid, when a mathematical formula will yield you better results for less effort. If it's the latter, sticking to natural language is stupid too, because the whole system is useless - if there are no patterns to study, you may just as well flip a coin or read from /dev/urandom.
Now, the very existence of prompt engineering tells us we're likely dealing with the first case - with understandable patterns. However, our systems are not black boxes. Prompt engineering is, at its best, turning interactions with LLMs into an empirical science, which makes no sense when dealing with human-made artifacts. We don't need to discover the patterns, we can read them off the thing, and we can adjust the thing to manifest different patterns.
> It's like trying to write production code in an esolang like Malbolge.
It's more like trying to learn programming via scientific method: running sets of random characters through the compiler, evaluating output, making a hypothesis, running more random strings through the compiler, checking if that proves or disproves the hypothesis, and adjusting the next iteration to generate slightly less random character strings - rinse, repeat. Going through all that effort is stupid, because you could just pick up a book instead - programming is a man-made job, and all the rules are designed in.
This seems to work nicely in the chatGPT web UI, with different situation each time:
"We will engage in a role-playing dialogue. The dialogue will take place in turns, starting with you. Always wait for my response. Use a conversational, informal, colloquial style. Try to use simple English, so that a learner of English can understand.
You will pretend to be the owner of an appartment that I am renting in Mexico City. Pretend to be an unpleasent and unreasonable person. Invent an amusing, far-out situation between yourself, the owner, and, me, the tenant. First explain the situation and then allow me to respond."
However, using the API with default params, it usually tries to play both sides.. there's seems to be a difference, any ideas?
Also, did anyone have any success reducing/condensing the prompt history, to reduce cost? Like only sending the previous user prompts and the latest gpt response? Or, using gpt to summarize previous dialogue?
ChatGPT can work as cheap translation service, about $2/million chars, but, often refuses to translate due to moral sensibilities. :D
Like COBOL. Except that this is worse than COBOL in every single way ^^