Hacker News Reader

Comment by devmunchies | Hacker News Reader

Many think prompt engineering is just like being good at writing Google searches, where my job is to be good at knowing "how Google/GPT4 thinks"—good at writing a single query.

However, I think prompt engineering will evolve to be an actual technical role, akin to Data Engineering (the people who make the systems, pipelines, ETL jobs, etc for the data).

Prompt engineers will build systems that facilitate prompt generation. Meaning that prompts will be dynamically generated or at least partially generated with modifications or additions to the raw user prompt.

It's the difference to being able to write HTML vs being able to do all the backend work to dynamically generate the HTML for my Amazon homepage (including the performance benchmarks and other strategic requirements), for example.

View on HN

I don't think it will evolve at all, because models will just become better at understanding what people mean. There's no point in trying to be a better prompter as a competitive advantage.

Already, chatGPT and bing can both give great on-topic answers to 3 word queries. and the fact that you can infinitely refine it is great

OTOH i think there is space for developing GUIs for prompts. Makes them more engaging

> Already, chatGPT and bing can both give great on-topic answers to 3 word queries. and the fact that you can infinitely refine it is great

You’re completely ignoring the system prompts that OpenAI/Bing have already set up so your “3 word query” works as you intend. These system prompts are what prompt engineering is all about.

these prompts by definition do not need engineering because they work for all cases all the time. and they can only get better over time

Huh? How would they get better without engineering? Why would it be a good idea to use them in all cases all of the time?

There are exactly 2 of those prompts in the whole world

I bet they took a couple tries to come up with them.

Here's an early version of the Bing prompt. Last I heard it was already on version 96, and that was a few weeks ago: https://simonwillison.net/2023/Feb/15/bing/#prompt-leaked

Disagree. Some examples.

If I have an LLM in a video game that generates NPC dialog, I might want to feed more information to the prompt based on things my character has done in the game so the NPC dialogue is more relevant to me.

Maybe I want to inject the user’s location or the current weather in Sunnyvale, CA for the specific query. Or maybe I want to inject that the user is currently at Disneyland.

Maybe the user really likes a specific tv show and I want to let the model know that. Or maybe the specific TV show was DMCA’d by an IP owner and I need to put that in the prompt.

Do I need to detect that the user is below 13 yrs old and use a different prompt (or a different model altogether)?

Maybe I want the model to only ever respond with JSON without the user needing to specify. It needs to be clarified in the prompt with no way for the user to override it.

Models can be simplified to: input to output. Prompt engineering will be engineering the inputs.

Another example. Someone has yet another new programming language. Expected infrastructure was github, website with docs, editor/IDE integration, community repo, slack/mailinglist, etc. "Now" it's also a "chatoverflow" capable of answering "how do I ...?" and "translate this <code in popular language>". What does a MVP training set for this look like? Can translation be pushed to permit assimilating "batteries" from similar languages? What does the infrastructure for that look like? Does this mean a half century of agonizingly slow language evolution is about to state change?

By the time you have finished writing the job description for this prompt engineer, you have your prompt ready.

I’ve been doing the same thing with a number of projects, building chains of prompts from one api call to another e.g. for ConjureUI (self-creating, iterable UIs that come into existence, get used, then disappear) https://youtu.be/xgi1YX6HQBw how it works to generate a full self-contained react component:

1. Take user task

2. Pass it to a prompt that requests a Product UI description of a component

3. Pass 1+2 to another that asks for which npm packages to use

4. Pass 1+2+3 to a templated prompt to write the code in a constrained manner

5. Run 4 in a sandbox to see if there are errors, if so pass it back to #4, looping

It’s currently quite slow, but that’s an implementation detail I think.

> 3. Pass 1+2 to another that asks for which npm packages to use

I see a fresh new generation of supply chain attack, or more prompt engineering to hopefully filter out malicious packages

Once the malicious package is added to the universe of acceptable packages, it doesn't matter much. Prompt engineering is not a solution you that.

Yes, that wasn't a priority here, but I also don't think it's much of a concern with e.g. GPT-4's `system` vs `assistant` vs `user` roles. Would be another thing to work on, but nothing worth doom and gloom.

Although, 'script(/injection) kiddie' will be an interesting phenomenon in the future...

You can probably feed a curated list of allowed packages for this step

It's macro substitution on crafted templatized prompts at best. Calling it engineering dilutes the term either further than it already is. (Looking at you Salesforce "Engineer").

The term is already maximally diluted, but it is nice that coders are getting a turn at trying to gatekeep it.

Yeah. Software people getting uppity about use of the word “engineer” is…a hilarious thought. You made your bed on this one.

Engineers Canada has recently won a case against a software developer that used the term, “engineer” in their profile.

Here are a couple of good resources around prompt engineering

https://www.promptingguide.ai/

https://lilianweng.github.io/posts/2023-03-15-prompt-enginee...

These clearly demonstrate that there's engineering skills beyond "I want you to act as a Linux terminal."

Also note that there's a difference between the prompt that you use on ChatGPT and the one that you use in the GPT API or other LLMs. In the first, you're already dealing with the prompt that openai supplied to the LLM in the first place.

I stopped reading your second link after seeing that the person teaching prompt engineering demonstrated a lack of understanding regarding proper sentence structure.

According to their linkedin, the author’s job is “Head of Applied AI Research at OpenAI”.

Right, they use slightly imperfect syntax - because they're a non-native speaker.

Is that what you discount them for?

No. Of course that's not the reason. Please don't take your biases and make those sorts of statements about what you believe is in somebody else's thoughts.

The reason was what I stated, which has nothing to do with the provenance or capabilities of that person.

But why call it prompt engineer and not just data or, you know, software engineering? Prompt engineer means the prompting, and that’s already quite dead as it is: you can just ask gpt4 to fix the user input for a specific goal. We do this in our pipeline and it’s obviously (much) faster and cheaper than ‘prompt engineers’, but it is usually also simply better for what we need it for.

The stuff you say it will evolve in already exists and has names everyone uses.

Feature engineering [1] is probably the closest existing analogue.

[1]: https://en.wikipedia.org/wiki/Feature_engineering

I have an idea what skills are required to dynamicly generate HTML and how to measure quality.

Can you share something similar for "prompt engineering" ?

See my other comment I just posted to someone else for examples.

So you say "feed", "inject", etc.. but these just alternative verbs for "describing". What skills are needed? Where is the ingeneering part?

Well, it depends on what you need from your prompt.

What engineering is required for Amazon to deliver me an HTML page? Are the backend engineers just “injecting” HTML? Of course not. It’s not the same as generating the HTML for a personal blog.

As far as what skills are need, it depends on how any given team wants to modify the prompt. It’s up to the ingenuity of the engineers. Maybe they need to hit 5 other models before generating the prompt. IDK. The prompt may change for any given scenario, which is why you need engineers to build the system.

I'm working on a prompt engineering product at a stealth mode startup. Would it be possible for me to pick your brain sometime? I'd like to understand more about your workflow, and how our product could fit into that.

No pressure. My email is in my profile.

Crucially important that engineers who are building systems like that that work by concatenating prompts together have a very solid understanding of prompt injection attacks.

Lol

What’s funny? Try to write a chat bot built on an LLM that adapts to your personality over time and do it without EVER dynamically changing the prompt.

Reply on news.ycombinator.com