Prompt engineering playbook for programmers
addyo.substack.com
addyo.substack.com
- In Context Learning (providing examples, AKA one shot or few shot vs zero shot)
- Chain of Thought (telling it to think step by step)
- Structured output (telling it to produce output in a specified format like JSON)
Maybe you could add what this article calls Role Prompting to that. And RAG is its own thing where you're basically just having the model summarize the context you provide. But really everything else just boils down to tell it what you want to do in clear plain language.
The second phase can be with a cheap model if you need it to be.
Pseudo-prompt:
Prompt 1: Do the thing, describe it in detail, end with a clear summary of your answer that includes ${THINGS_YOU_NEED_FOR_JSON}.
Prompt 2: A previous agent said ${CONTENT}, structure as JSON according to ${SCHEMA}.
Ideally you use a model in Prompt 2 that supports JSON schemas so you have 100% guarantee that what you get back parses. Otherwise you can implement it yourself by validating it locally and sending the errors back with a prompt to fix them.
Be clear with your requirements. Add examples, if necessary. Check the outputs (or reasoning trace if using a reasoning model). If they aren't what you want, adjust and iterate. If you still haven't got what you want after a few attempts, abandon AI and use the reasoning model in your head.
“You are an expert doctor, help me with this rash I have all over” will result in a fairly useless answer, but using medical shorthand — “pt presents w bilateral erythema, need diff dx” — gets you exactly what you’re looking for.
Lay person’s description -> translate into medical shorthand -> get the expert response in shorthand -> translate back.
Liability and error is obviously problematic.
When the LLM thinks _you_ wrote something, it's nice about it, and deferential. When it thinks someone else wrote it, you're trying to decide how much to pay that person, and you need to know what edits to ask for, it becomes much more cut-throat and direct.
I wouldn't be surprised if in a few years from now some sort of actual formalized programming language for "gencoding" AI is gonna emerge.
[1]https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
Example: Consider what PDB has done for our understanding of protein folding, as well as the ML/computational techniques they’ve enabled (eg, Alphafold). Most scientific questions and properties are not as data-rich as protein folding. What if they could be?
Longer answer: The last 15 years in computational/ML + science have shown that structured databases open up entirely new frontiers in discovery (eg Protein Data Bank, Materials Project). But most scientific topics/properties are NOT in structured DBs, they’re scattered about in millions of papers. It’s especially a huge problem in some topics in materials science. It’s not that these problems are data scarce, but that it’s hard to actually collate their data in a structured format. You literally cannot use most ML methods because structured DBs do not exist.
This paper is a way to generate massive structured databases of specialized, intricate, and hierarchical knowledge graphs from scientific literature. Fine tuning works, prompt engineering does not (at the time, perhaps this has changed). Once you have a database, you can analyze an entire subfield or topic in science with ML or stats methods.
Start out with Typescript and have it answer data science questions - won't know its way around.
Start out with Python and ask the same question - great answers.
LLMs can't (yet) really transfer knowledge between domains, you have to prime them in the right way.
I’ve also found it’s very good at wrangling simple SQL, then analyzing the results in Bun.
I’m not doing heavy data processing, but so far, it’s remarkably good.
However, asking it to implement "that thing that groups data points into similar groups" needs a bit more context (I just tried it) as K-means is very much specific to machine learning.
initial prompt: Start a new typescript file. It will be used for data science purposes
second prompt: Implement that "that thing that groups data points into similar groups"
The output was a full function implementing K-means (along with a Euclidean distance function).
https://chatgpt.com/share/68420bd4-113c-8006-a7fe-c2d0c9f91d...
Doesn't this ruin / ignore the point we're discussing? I don't think anyone thought otherwise?
Every day tech broism gets closer to a UFO sect.
Obviously how exactly they work still isn’t fully explained, but calling basic principles magical is too far in my opinion.
maybe to gp but i thought no one knows how they work.
Make prompts focused with explicit output with examples, and don't overload the context. Then the 3 you said basically.
As an exercise for the reader, I encourage you all to try the examples vs. control prompts in prompt engineering papers for chain of thought prompting, and you’ll see that the latest models have either been trained to or instructed to reason by default now - the outputs are close enough to equivalent.
CoT prompting was probably much more effective a few years ago on older, less powerful models.
You may find some benefit in telling it exactly how you want it to reason about a problem, but note that you may actually be limiting its capabilities that way.
I’ve found that most of the time, I will let it use its default reasoning capabilities and guide those rather than supplying my own.
My usage has converged to making very simple and minimalistic prompts and doing minor adjustments after a few iterations.
I've found that with more advanced prompts, the generated code sometimes fails to compile, and tracing the issues backward can be more time consuming than starting clean.
I also feel the need to caution others that by letting the AI write lots of code in your project it makes it harder to advance it, evolve it and just move on with confidence (code you didn't think about and write it doesn't stick as well into your memory).
My experience as well. I fear admitting this for fear of being labled a luddite.
At the same time, I’ve seen the system prompts for a few agents (https://github.com/x1xhlol/system-prompts-and-models-of-ai-t...), and they are huge
How does that work?
Maybe a super salty take, but I personally haven't ever thought anything involving an LLM as "proper engineering". "Flailing around", yes. "Trial and error", definitely. "Confidently wrong hallucinations", for sure. But "proper engineering" and "LLM" are two mutually exclusive concepts in my mind.
This is even worse than "software engineering". The unfortunate thing is that there will probably be job postings for such things and people will call themselves prompt engineers for their extraordinary abilities for writing sentences.
Without it, the chances of getting to a solution are slim. With it, the chances of getting to 90% of a solution and needing to fine tune the last mile are a lot higher but still not guaranteed. Maybe the phrase "prompt engineering" is bad and it really should be called "prompt crafting" because there is more an element of craft, taste, and judgment than there is durable, repeatable principles which are universally applicable.
You're not talking to managers here, you can use plain english.
> Maybe the phrase "prompt engineering" is bad and it really should be called "prompt crafting" because there is more an element of craft, taste, and judgment than there is durable, repeatable principles which are universally applicable.
Yes, the biggest problem with the phrase is that "engineering" implies a well-defined process with predicable results (think of designing a bridge), and prompting doesn't check either of those boxes.
I have been using LLMs very specifically to drive towards those goals, and prompt engineering (or crafting given you dislike the phrase "engineering") is a crucial tool to get to those outcomes. And yes, sometimes that means writing my own code itself to interact with them, template prompts, post process, utilize them in workflows. And so, the more I think about it, the more that I see patterns in how I create prompts (that probably could be automated with the same tools used for content templating) that make it feel somewhere in the middle of engineering and crafting.
My guess is that semantic ambiguity makes a lot of folks uncomfortable. If that's engineering, what isn't engineering, and doesn't it dilute engineering? Yet from the other angle, it's absolutely the case that LLM utilization is absolutely becoming a greater and greater part of how cutting edge companies write code, so much so that the "code" that the humans must put effort and intention into writing is just as much prompts as it is the fine tuned artifact created by running the prompt. If the reality of what actually is "code" is itself evolving, it's hard to imagine anything but that what constitutes software engineering must also itself be evolving in a very fundamental way.
Since what's proper and meaningful depends on a lot of variables. Testing these, keeping track of them, logging and versioning take it from "vibe prompting" to "prompt engineering" IMO.
There are plenty of papers detailing this work. Some things work better than others (do this and this works better than don't do this - pink elephants thing). Structuring is important. Style is important. Order of information is important. Re-stating the problem is important.
Then there's quirks with family models. If you're running an API-served model you need internal checks to make sure the new version still behaves well on your prompts. These checks and tests are "prompt engineering".
I feel a lot of people take the knee-jerk reaction to the hype and miss critical aspects because they want to dunk on the hype.
On the other hand, prompt tweaking can be learned in a few days just by experimenting.
Not this one.
> (alt) a field of study or activity concerned with modification or development in a particular area. "software engineering"
This one ^^^
Too many people seem really triggered by this. I don't know why, but it's weird. It's just a term. It's well understood by now. The first 5 pages on google all state the same thing. Why bicker about something so trivial?
That could be said about ordering coffee at local coffee shop. Is there a "barista order engineering" we are all supposed to read?
> Re-stating the problem is important.
maybe you can show us some examples ?
The way I see it, it's a bit like putting up a job posting for 'somebody who knows SSH'. While that is a useful skill, it's really not something you can specialize in since it's just a subset within linux/unix/network administration, if that makes sense.
For the uneducated, law engineers are members of the Congress / Parliament / Bundestag / [add for your own country]
I don't think you have to worry about that.
Let's say you have two teams of contractors. One from your native country (I'm assuming US here), working remotely and one from India, located in India.
Would you communicate with both in the exact same manner, you wouldn't adjust your messaging in any way?
Of course you would, that's exactly what "prompt engineering" is.
The language models are a different and a bit fiddly at the moment, so getting quality output from each requires a specific input.
You can try it yourself, ask each of the big free-tier models to write a simple script in a specific language for you, every single one will have a different output. They all have a specific "style" they fall into.
Many people just won't put any conscious thought into trying to get better on their own, though some of them will read or watch one thing on the topic. I will readily admit to picking up several useful tips from watching other people use these tools and from discussing them with peers. That's improvement that I don't think I achieve by solely using the tools on my own.
And documenting the current state of this space as well. It's easy to have tried doing something a year ago and think they're still bad.
I also usually prefer researching some area before reinventing the wheel by trial/failure myself. I appreciate when people share what they've discovered with their own their time, as I don't always have all the time in the world to explore it as I would if I were still a teen.
It may seem simple, but in my experience even brilliant developers can miss or misinterpret unstructured requirements, through no fault of their own.
Given input (…) and preconditions (…) write me spark code that gives me post conditions (…). If you can formally specify the input, preconditions and post conditions you usually get good working code.
1. Science of programming, David Gries 2. Verification of concurrent and sequential systems
I'll love to be wrong though. Please share if anyone has a different experience.
I think we'll also see ways of restructuring, organizing, and commenting code to improve interaction with LLMs. And also expect LLMs to get better at doing this, and maybe suggesting ways for programmers to break problems down that it is struggling with.
I get by just fine with pasting raw code or errors and asking plain questions, the models are smart enough to figure it out themselves.
Many prompt engineers do measure and quantitatively compare.
Maybe this is why we disagreed on a previous comment. I think guess and check is generally the best way to solve problems, so long as your checks well-designed (which to be fair, does require understanding the problem) and you incorporate the results from the check back into further guesses, and you can afford to brute force it (which is statistically the most common problems big and small I've had to solve).
In a lot of ways, there's nothing fancy about that, it's just the scientific method -- the whole point which is that you don't really need to "know" about the problem a priori to get to the truth, and that you can recompile all dependent knowledge from scratch through experimentation to get to it.
In practice it feels really expensive but it's also where the best insights come from -- experimentally re-deriving common knowledge often exposes where its fractures and exceptions are in clear enough detail to translate it into novel discovery.
When it comes to "prompt engineering", the argument is even less compelling. Its like saying typing in a search query is engineering.
> Calling someone a prompt engineer is like calling the guy who works at Subway an artist because his shirt says ‘Sandwich Artist.’
All jokes aside I wouldn't get to hung up on the title, the term engineer has long since been diluted to the point of meaninglessness.
There are prompts to be used with API an inside automated workflows and more to it.
They were also unaware of the fact that if you create a filename with spaces you must then escape/quote it for it to work.
They requested this important information to be included in the user manual (the users being sysadmins at very large companies).
Actually our sales INCREASED when they fired like 100 of these guys :D
Edit: I also love that the examples come with "AI’s response to the poor prompt (simulated)"
Meanwhile, I just can't get over the cartoon implying that a React Dev is just a Junior Dev who lost their hoodie.
There is no common sense with it - it is just an illusion.
Example. "Always look when you cross the road" is a snippet common sense, with lack of heeding to that sense resulting in you potentially being hit by a car. Even a 4 year old wouldnt need the latter explanation, but most people could articulate that if they needed to. Its just a way of speeding up communication
A colleague and I were lamenting a laughably bad outage that we thought showed a total lack of common sense resulting in an obvious issue. Then we both realized that the team had never had such an experience whereas the two of us had. Every member of that team now has that particular “common sense”.
Likewise, “don’t run in front of cars”. As a kid, a mate broke his leg running onto the street and getting hit. I think near misses happen a lot when we’re kids.
But far fewer has an “common sense” about prompt engineering because there’s just much less experience.
There aren't enough software engineers to create the software the world needs.
I think you mean "to create the software the market demands." We've lost a generation of talented people to instagram filters and content feed algorithms.
Like writing out clear requirements.
You can also make the AI review itself. Have it modify code, than ask to review the code, than ask to address review comments, and iterate until it has no more comments.
Use an agentic tool like Claude Code or Amazon Q CLI. Then ask it to run tests after code changes and to address all issues until test pass. Make sure to tell it not to change the test code.
It also allows me to more easily control what the llm will do and not end up reviewing and throwing 200 lines of code.
In a nextjs + vitest context, I try to really outline which tests I want and give it proper data examples so that it does not cheat around mocking fake objects.
I do not buy into the whole you’re a senior dev etc. Most people use Claude for coding so I guess it’s engrained by default.
Like what Agentic IDEs are starting to do. I don't copy paste code in the correct way to optimize my prompt, I select the code I want, with MCPs picking up you might not even have to paste input/output the Agent can run it and parse it into the LLM in an optimal way.
Of course, the quality of your instructions matter but I think that falls outside of "prompt engineering"
-- https://www.theregister.com/2025/05/28/google_brin_suggests_...
It is fascinating that you can give instructions as if you were talking to someone, but part of me feels like doing it this way is imprecise.
Nevertheless, having these tools process natural language for instructions is probably for the best. It does make these tools dramatically more accessible. That being said I still feel silly writing prompts as if I was talking to a person.
But I am skeptical over the idea that asking a LLM to be an expert will actually improve its expertise. I did a short test prompting ChatGPT do be an "average developer, just smart enough not to get fired", an "expert" and no persona. I got 3 different answers but I couldn't decide which one was the best. The first persona turned out quite funny, with "meh" comments and an explanation about what the code "barely" does, but the code itself is fine.
I fear that but asking a LLM to be an expert, it will get the confidence of an expert rather than the skills of an experts, and a manipulative AI is something I'd rather not have.
About the roles: Can you measure a difference in code quality between the "expert" and the "junior"?
> The key is to view the AI as a partner you can coach – progress over perfection on the first try
This is not how to use AI. You cannot scale the ladder of abstraction if you are babysitting a task at one rung.
If you feel that it’s not possible yet, that may be a sign that your test environment is immature. If it is possible to write acceptance tests for your project, then trying to manually coach the AI is just a cost optimization, you are simply reducing the tokens it takes the AI to get the answer. Whether that’s worth your time depends on the problem, but in general if you are manually coaching your AI you should stop and either:
1. Work on your pipeline for prompt generation. If you write down any relevant project context in a few docs, an AI will happily generate your prompts for you, including examples and nice formatting etc. Getting better at this will actually improve
2. Set up an end-to-end test command (unit/integration tests are fine too add later but less important than e2e)
These processes are how people use headless agents like CheepCode[0] to move faster. Generate prompts with AI and put them in a task management app like Linear, then CheepCode works on the ticket and makes a PR. No more watching a robot work, check the results at the end and only read the thoughts if you need to debug your prompt.
[0] the one I built - https://cheepcode.com
What a world we live in.
Regardless, I think that programmers are quite well-suited to the methods described in the article, but only in the same way that programmers are generally better at Googling things than the average person; they can imagine what the system needs to see in order to produce the result they want even if that isn't necessarily a natural description of their problem.
When I say that your native language is now a programming language, I mean that you are now the Project Manager, and the code is generated automatically. But because the code is auto-generated, the focus shifts towards the “prompt” as the first and potentially only step in generating code.
Whether its true or not it is besides the point - its getting kind of boring and it seems to be drowning everything else. It feels at least from the outside like a profession that used to require intelligence/skill, and something creative problem solving just went downhill fast. The anti-SWE crowd (e.g VC's, AI devs, CEO's, etc) seems to be winning the argument at the moment as to what the future will bring in these forums. Every podcast I hear its always software engineering that is mentioned as the industry to be disrupted away.
Who would of thought the job of the future only 5 years ago would be the first to go? Hope I'm wrong, but it seems the AI crowd's main target atm.
WHICH IS IT?