So long “prompt engineering,” we hardly knew ya
medium.com
medium.com
---
* diagnosis
* decomposition
* reframing
* constraint design.
Diagnosis is discovering the problem that AI can solve. This is the human part of knowing that a problem exists. Learning to ask the right questions, look at the different ways that the problem can be seen.
Decomposition is about splitting the big problems into bite-sized ones. Take the problem apart, examine it, and let AI help you determine your findings since it handles data so well. Instead of tackling the biggest problem, take it apart and work on the smaller parts to achieve small successes.
Reframing is about shifting your perspective and seeking new interpretations. Extrapolating and recombining the parts of the problem in order to identify the meta components. Perhaps a new way of looking at the problem may find a solution hidden in plain sight.
Constraint design is about setting boundaries for the solution. Knowing what to accomplish, and when to know it is done. Setting the length, style, and description of the audience can help AI understand its mission. But we have to know that first in order to instruct.
---
As someone who asks GPT and junior developers for lots of things, there are a lot of similarities. I don't imagine that is going away, at least until we wire LLMs up to a huge amount of rapidly changing, cross-silo context so it could understand "Fix the monitoring that slowed our recognition of yesterday's bug". So being thoughtful isn't going away. The author agrees with that (see above), but doesn't make clear where he draws the boundary between "being thoughful" and "prompt engineering"
IMO - Folks seem to be reacting to the term “engineer” in the same way traditional engineers with engineering exams do to software engineers claiming to be engineers despite often graduating from a LAS school or not at all - I have a bunch of PE EE in my family and they treat my claim to be an engineer with extreme scorn.
Carpenter -> Wood engineer
Tailor -> Fabric engineer
Ceramist -> Clay engineer
See how dumb it looks? And maybe the field is too new to have a specific word for it, but calling it "engineer" diminishes the level of actual engineers.
When you say diminished, are you are talking from a societal point of view?
Prompt Engineering is a fun title in a fast moving and interesting field. Everyone's taking it far too seriously.
I'm going off to become a Clay Engineer, that sounds fun!
I personally think it’s worth reserving certain titles for qualified practitioners. If you want to call yourself a Doctor, Lawyer, or Engineer, I think that should mean something.
In many countries it's more a title associated to education and specific degrees from specific schools than a license to practice some professional organisation could revoke.
While, from my limited knowledge, doctors and lawyers both seem to have some sort of license and controlling body in most, if not all, countries.
There are quite a few jurisdictions where some Engineer title is protected:
https://en.wikipedia.org/wiki/Regulation_and_licensure_in_en...
The UK seems to discriminate on a per discipline basis. Canada apparently is ambiguous, with self-regulating bodies but courts dismissing cases regarding job titles. Germany has one, but only for civil engineers (still according to the wikipedia page).
Still only relying on this wikipedia page, there are on the other hand many countries where although the title is protected, it simply requires one to have studied a certain number of years (Poland), to have completed a specific degree (Brazil, Chile, Germany), or a specific degree in one of a few select higher-education schools (France, Turkey).
Go this way, instead:
Engine-maker Road-maker Trend-maker Fabric-maker Clay-maner Wood-maker Makeup-maker Book-maker Law-maker
Each school of making shall also have ranks: Vice Chief of Book-making, Novice Fabric-maker, Treasurer of Makeup-making et cetera
If software development (or even "programming"!) is the same, the same thing will happen. If not, any other words we steal will lose their shine too and engineers will start calling themselves something else to get it back.
Outside of tech, being an actual engineer involves universal standards, certifications and the onus and responsibility of failures that the deregulated libertarian paradise of the tech world is impossible to implement.
Actual engineers go through rigorous testing, certification and have universal standards that uphold them to meet these standards regardless of business pressure.
Until the tech and software world have anywhere near that level of scrutiny by public institutions (good luck, all you will hear is the screams of "communism!") then frankly its already looked dumb ever since developers were even called "engineers" to begin with.
It is very rare I think for completely new words to emerge nowadays.
Prompt engineer is descriptive and you can guess its meaning.
if you want a long-lasting career in AI you need to work on the actual AI stuff, not just using the AI stuff.
Back in the day, when those new fangled relationship database things came on the scene, do you think people would have been well advised to try and find work on the actual database engine itself, instead of the more frivolous work of using the new technology to, say, solve actual business problems?
If you are not rigorous, then what you are doing is essentially "black art". It may work for some tasks ad-hoc, but with the rapid pace of model improvement your skill will likely become irrelevant/not needed quickly.
You need some amount of experimentation to get the best results but in my experience what works for one model does nothing or worsens the output in others. Adding loras and different types of images into the equation makes this so variable that I would never consider it useful besides keeping a few key words I used to get x good result on y model and experimenting with those when I start a new project.
Calling it "prompt engineering" seems odd.
I don’t think LLM offer standardized API like that ?
I think this is more for people, though, that want to maximize the use of AI in their own field, isn't it? The "knowledge worker enhancer"?
I am not sure they really need to work on the actual AI stuff...
The latter latches on to the surface level of any new tech but proceeds with swagger befitting a Nobel laureate.
Prompt engineering is not the same as programming. In some regards, it is “better”. In other regards, it is “worse”. They’re two different yet similar disciplines, each with their own strengths and weaknesses, but both equally legitimate.
Programming is simply the skill of being able to communicate with a compiler/interpreter effectively.
Prompt engineering is simply the skill of being able to communicate with an LLM effectively.
I wish we weren’t so divided over an incredible new technology.
But, even today prompts and rules of writing are not transferable even between existing LLMs. Future LLMs will have different architectures and requirements. I suspect today's prompt will be split into data and prompt. Probably just references, keywords to actual data.
In other words current state is transitory, next will be very different.
https://www.reddit.com/r/datascience/comments/14nbwfv/where_...
I mean, you think those graphics optimizations we pour millions of dollars into before releasing AAA games will matter when the GTX 8020 outperforms a 4090?
Hint: Delivering value for actual people is rarely the result of sitting on your hands and waiting for the next big platform, or even rolling up your sleeves and trying to learn how to build the next big thing.
You have this idea that by investing energy in something that will be obsoleted you're losing out, but spoiler, that's how 99% of software that delivers actual value works.
The cutting edge of tech almost always ends up being PaaS/SaaS serving itself:
Your mail gets to you because someone is working on software with limitations we solved decades ago.
Your paycheck ends up in your bank account because people invest a ton of time in codebases subject to problems we solved long ago.
Your anti-lock brakes aren't built on a Rust codebase, but some horrible memory unsafe mess running on a processor that's a decade out of date.
—
The reality is: 99% of the effort that goes into trying to build the next big thing goes nowhere. The expected value of you trying to learn "the actual AI stuff" to the greater world is near 0 compared to you "just prompt engineering" and putting out something that solves a pain-point nicely with GPT 3.5.
At the end of the day most of the value that gets delivered to actual users comes from engineers who went deeper into extracting value from the current thing.
We still need people to work on the next big thing so that the 1% of effort that isn't wasted can actually materialize... but in my experience the most successful engineers in that regard are still able to realize the delusion it requires, without being paralyzed by the cognitive dissonance that realization invites.
a different analogy that gets at my original concern: becoming an expert prompt engineer for an particular LLM is like becoming a power user for a piece of proprietary software that isn't getting any more updates.
In tech we take it for granted that just because there's some new hotness everyone wants to jump on it day 1. GPT 5 could drop tomorrow and if your tool delivers value using 3.5, it's not going to magically stop delivering value, and in most verticals people will prefer your battle tested 3.5 to some brand new 5.
And if 5 does simplify prompts for your use case and there'll still only be two options:
- making a the same thing as what you made with 3.5 is now trivial... in which case you still have the mindshare and the distribution solved to a degree your newly enabled clones don't.
- making a better version of what you made is now trivial... in which case you can just as trivially improve your version and already have the mindshare and distribution solved.
At the end of the day software developers often struggle to fit software into the larger ecosystem it slots into before it becomes something valuable, and to be GPT has been an amazing case study in the fact.
I think the use of the word engineering did half the damage, and I think people confusing the twitter memes with the interesting attempts at prompt engineering (via ReACT, Gorilla, etc) did the other half, but at the end of the day I think a lot of people will be left kicking themselves when "misguided prompt engineers" end up solving real useful problems in ways they didn't think were possible well before we reach the arbitrary goalposts for foundation models that people keep setting up.
- a C developer that writes modem drivers at a telecom company in the 90s about html/perl developers building first interactive websites
Is probably what someone said many decades ago.
Building computers, operating systems and compilers was always a niche.
So if you're a computer scientist or an electrical engineer, sure.
But if you're not, there was, is and will be "power users" who make the most of it.
- Teach, fix shit and bridge the gap for mortals
- Enjoy technology with a deep but not foundational understanding.this is different from LLMs because they don't have explicitly constructed abilities that we can compare across models. so we need to step back and approach every model as brand new and figure out what they're capable of. just because you have a prompt that works amazingly in one model, there's no guarantee that that prompt will continue to work in bigger "better" models.
so rather than working to devise way to trick a particular model into doing your task, it would probably be a better use of your time to learn how to train/modify models to explicitly solve the problem you care about.
Reading that article makes me wonder if we're even talking about the same thing.
1. API, concepts, etc docs with explanation. Much faster then googling then scrolling through tons of texts with blinking adds
2. Write simple things that I don't want to think about. Like in Python process all files in a directory, follow the links. Saves time.
3. Try things I don't know how to do. The recent was checking if user pressed a key without blocking in Python. Nontrivial, but possible. We went through several options till found the one which works on Ubuntu.
So, it's useful, no regrets about subscribing. Funny thing I'm using it working on toy GPT
I have used several of 'the big' SaaS GPTs, and have gotten great use from them.
There is absolutely a use for these tools, but they do take some small amount of skill to get good results.
It all comes down to the context you can provide to steer the answer to what you really need. The better you can describe your current state and what you want from the answer, the better the answer will be.
My only use so far is get inspiration for type naming, and very simple scripts that i'm too lazy to write myself.
But this happens less than once a month.
The latest ChatGPT model, gpt-3.5-turbo-0613 has better system prompt steerability, and with some prompt engineering I can get GPT-4 quality results out of it at a fraction of the cost.
Ultrapopular tools like LangChain and AutoGPT are essentially just prompt engineering under the hood.
Where are they now?
I think the big problem is actually finding problems you could use LLMs in. I think anyone that's played with them tends to have a good guess at whether they could or couldn't do something (although you need to really test to be sure, do some prompt engineering etc), but actually finding problems to work on that are within the realm of being solvable and useful is the hardest part imo.
Other industries have proper certifications for "engineering" that I.T. absolutely doesn't, yet someone who figured out how to center a div calls themselves a software engineer.
https://www.theatlantic.com/technology/archive/2015/11/progr... <- related article.
I reckon if you're going to give devs the right to call themselves engineers just because they write code to solve business problems, you don't have much of a leg to stand to judge people whom write natural language into a software solution, and receive output that solves business problems.
I'm going to link my own snarky titled by actually okay-ish take on prompt engineering, emphasis on the actual engineering, to show that the author doesn't even know what prompt engineering is - https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
Prompt engineering is not designing a cool prompt. It's when you start applying genuine techniques to do things that are not possible with tokens alone. For example:
"What's the definition of {apple|orange}" where {apple|orange} is the mathematical average of those two words. This is prompt engineering. Right now, Prompt Engineering is basically in Stable Diffusion through Automatic1111, it's in libraries like microsoft Guidance or LMQL, and not a whole lot else.
1) That as tools get better, operator specialization is less useful. I understand how this feels like it makes sense, because for for binary tasks like driving it’s true, but for creative tasks, I struggle to think of a single example where this is the case. Which leads to
2) Humans won’t figure out how to use these tools in increasingly complex/weird ways to create increasingly complex/weird outputs.
I think in general, everyone has been so conditioned by the idea of singularity (which to be frank is a completely tangential concept to contemporary LLMs) that they refuse to see these things for what they are: tools built by humans to serve humans when operated by humans.
Add as many layers of “self-prompting” as you want there, but a human still set the original intention and they will be the ones to judge the ultimate outputs.
I think one key difference is that (most) programs that people are used to writing tend to be deterministic (when not intentionally random...) whereas LLM prompts pretty much always end up with an nondeterministic output.
Where I DO think Prompt Engineering goes away is because codegen and code orchestration of LLMs rises up to take its place. Hence Prompt Engineer -> AI Engineer https://www.latent.space/i/131896365/the-role-of-code-in-the...
Clear away the faff and ignore the “AI is improving sooo fast ohh what a magic” remarks. What you’re left with is a tepid article about prompt engineering.
In a couple of years, perhaps it will come to be known as “machine psychology”?