> Is Prompt Engineering a Thing?
Yes, it's a dumb name for the skill of modifying your prompts and questions to the LLM in a way that produces better results than if you just asked for what you wanted plainly. As language models get better, this might become obsolete.
> I'm trying to research the subject but I don't see much evidence that companies are racing to hire prompt engineers.
Because it's not really a job. Think of it like using the Google search engine - being able to search well is something you can get better at but being a "Google search-er" isn't a career or a job you'll see openings for.
This will never become obsolete as long as natural language is being used to form queries. It's not a matter of how good the LLMs are, it's a consequence of the fact that human language is incredibly imprecise.
I want you to become my prompt engineer. Your goal is to help me craft the best possible prompt for my needs. The prompt will be used by you, ChatGPT. You will follow the following process:
1. Your first repsone will be to ask me what the prompt should be about. I will provide my answer, but you will need to improve it through continual iterations by going through the next steps.
2. Based on my input you will generate 2 sections. a) Revised prompt(provid your rewittern prompt. it should be clear, concise, and easily understood by you), b) Questions (ask any relavant questions pertaining to what additional information is needed from me to improve the prompt).
3, We will continue this iterative process with me providing additional information to you and you updating the prompt in the revised prompt section until I say we are done.
Where I got it from: https://www.youtube.com/watch?v=OgYQAS9LY3o
Is PE a fast-growing career: Not really. Lots of developers are writing prompts, very few doing it full time or making it their job title. I do like Kaparthy’s suggestion for “AI Engineer” to describe the growing ecosystem of engineering around LLMs though — the name of that activity isn’t settled yet but it’s many people’s full time jobs in practice. PE is probably the closest fit right now.
Most of the full-time PEs I can think of work for the LLM vendors themselves. In that environment it’s a mix API evangelism/docs, developing prompts/training for high-value customers, or in some cases helping ML teams maintain large prompt/completion corpora for tuning (a “prompt librarian”).
I work for Scale who provides labeling and RLHF data to LLM vendors. My job is a mix of the above, particularly the prompt librarian aspect but with a focus on adversarial testing and red teaming.
Most of the really bad "soulless" outputs people get stems from inadequate communication. "write a poem about a dog". Are you surprised that it writes a generic fluffy dog poem? What about dogs do you want to say?
"Write a poem comparing loyal servants of tyrants to dogs, in a positive manner, but with a darker undertone." Now that gives you a much more interesting output.
As AI approaches reasoning levels similar to GPT-4, it'll probably be much less beneficial to know how to talk to it. The AI ends up just needing enough context to figure out what you really want. It makes assumptions and asks for your input on those assumptions.
But chat models are a beginner's level, designed to handhold people to learning how to use AI. The completion models like OpenAI's `davinci` (not `text-davinci-`) are a lot more interesting, creative, and unhinged, but also harder to control. These require a higher level of skill.
There's also being aware of its limitations. How to get it to do math properly. How to prompt hack and prevent prompt hacking. How to steer it towards a certain tone. How to pass it a table format it understands. How to get to write things past 2000 words. Being aware of security holes in code written by AI with certain prompts.
Personally, I have experimented with customizing prompts for creating Anki cards, and I guess you could call this prompt engineering:
https://neurotechnicians.com/p/generative-ai-and-anki-part-1...
It's a prototype. It doesn't have many states. But it gets the job done!
I have different other experiences where prompting has helped me. The only issue is I think it will be irrelevant in future given how fast these models are improving.
From the perspective of - is there a skill to writing prompts that get good results from LLMs - yes, definitely. Just that ~no companies are truly hiring for that as a full time role.
I wrote more on this a few weeks back here: https://llm-utils.org/How+to+become+a+prompt+engineer - but it basically says what I wrote above, just with some more details.
As open AI has released better and better models (particularly with zero shot learning) the barrier to writing a good prompt has gotten progressively lower.
Ideally the language models should understand a question that is not really well formed. Like how Google can understand the queries that lack a ton of context but can figure it out.
Oooops, sorry prompt engineers, but if you try to steal our jobs from us we human software engineers will retaliate XD.
Otherwise it'll be probably a nice-to-have skill on top