A prompt pattern catalog to enhance prompt engineering with ChatGPT

While somewhat useful - there are no systematic ablation and comparison studies, datasets, and quantitative evals here. It's all anecdotal, so should probably be a blogpost and not a "paper"

Couple more that I found useful:

>List all the entities in the text that are ambiguous. Entities are ambiguous if you don't understand them, if they have multiple meanings and you can't decide which one from the context, or are phraseal terms whose meaning can be different in different cultures.

This allows the LLM to identify parts of previous prompt that the AI did not fully understand.

>List all the entities and relationships to other entities. Be precise, avoid duplicates, list each relationship only once.

This allows to compress the past context for longer tasks.

>Make a list of Google searches needed to build a factual answer, one sentence per search.

You get the list then

>For each sentence extract the main entity, decide whether to find more information about it on Google, wikipedia, ..., answer in json

This is my "ghetto" agent, it's not iterative, but it's stable enough to handle the unexpected

> It's the context enough to answer factually the user question? Answerer yes or no. Only answer yes if you are sure you can answer

To make the agent able to loop, or ask user for more context.

It's maybe interesting to think about a simple chat history over time giving some of the knowledge needed for improved interactions as related to the more complex patterns outlined here.

Here's a similar paper I ran across this morning: https://arxiv.org/abs/2305.18323. Github is here: https://github.com/billxbf/ReWOO

I indexed both documents with my own project, which uses semantic graphs to help with prompt assembly: https://github.com/FeatureBaseDB/DoctorGPT. DoctorGPT doesn't have dynamic prompt chaining yet, but I'm working on it. I hesitate posting any of the analysis of these papers using DoctorGPT here because it would be generated by the LLM, and not me...and some people seem to have an issue with that given this is a human forum.

My sense is that SKGs are important in refining questions, offering alternative approaches, managing context, reflecting on LLM responses, and more.

I think you shouldn't call it "prompt engineering" until you do some real "engineering". For example, if you use Guidance from microsoft, where you write a prompt "template" and let guidance filter the logits probabilities at the corresponding points for you, I claim that this is "engineering (the way we use it as SWEs anyway) because it's analogous to writing an actual program.

I was actually upset about everyone claiming they're a "prompt engineer" without any real engineering that I wrote a snarky github gist about it that was on the front page for a big. It's mainly pointing out that NLP can't even do real prompt engineering like Stable Diffusion folks can because we didn't build the right tooling for it yet.

https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

I recognize there's plenty of catnip here when it comes to calling this "engineering" or not, however, whatever you want to call it (prompt fiddling?), the techniques are crucial if you want to achieve reasonably consistent output from current-state LLMs. As models improve concerns about context window limitations will be reduced and it will be easier to discern user intent.

These are good straight-to-the-point guides:

- Prompt Engineering by BrexHQ: https://github.com/brexhq/prompt-engineering

- OpenAI guidance: https://help.openai.com/en/articles/6654000-best-practices-f...

- https://devblogs.microsoft.com/dotnet/gpt-prompt-engineering...

- (great examples): https://www.deeplearning.ai/short-courses/chatgpt-prompt-eng...

- (~hr) Karpathy talk: https://www.youtube.com/watch?v=bZQun8Y4L2A

tl;dr:

- Begin with the best and most forgiving model available, optimizing it later if necessary.

- Write your prompts in a clear, specific, and detailed manner

- Be mindful, however, of the tradeoff between including more detail and increasing latency or cost.

- If you're doing any complex reasoning, ask the model to show its work, help "stretch out" computation over more tokens

- "Escape" any included or quoted text properly to avoid confusing the model.

- Keep track of your token budget, considering the context window in conversations and response length.

- Employ an iterative process of measuring, adjusting, and improving your prompt engineering techniques.

Here is a more colorful formatted version of their recipe list.

https://docs.google.com/document/d/1G0TGB16lOf6hhGNUDnrM-kQy...

Not a permalink so recommend making your own local copy if you like it.

How much overlap is there currently between LLM prompt engineering and (for example) academic test/assignment question guidelines?

I can't wait until there is a standardized ChatGPT Bible to guide this emerging machine-priest caste. And these posts will be the equivalent of the dead sea scrolls.

Is it just me that thinks calling it "Engineering" is absolutely ridiculous?