Show HN: A simple ChatGPT prompt builder
mitenmit.github.io
Any Ideas/Suggestions are welcome :)
mitenmit.github.io
I was deeply involved in prompt engineering and writing custom prompts because they yielded significantly better results.
However it became tedious especially since each update seemed to alter the way to effectively direct ChatGPT’s attention.
Nowadays I occasionally use a custom ChatGPT but I mostly stick with stock ChatGPT.
I feel the difference in quality has diminished.
The results are sufficiently good and more importantly the response time with larger prompts has increased so much that I prefer quicker ‘good enough’ responses over slower superior ones.
It’s easier to ask a follow-up question if the initial result isn’t quite there yet rather than striving for a perfect response in a single attempt
I haven't noticed prompt size having an impact jut I'll test that.
Is it possible that you have a caching system too so that you are able to respond instantly with paragraphs of technical information to some types of requests that you have seen before?
As far as I understand, the earlier GPT generations required a fixed amount of compute per token inferred.
But given the tremendous load on their systems, I wouldn’t be surprised if OpenAI is playing games with running a smaller model when they predict they can get away with it. (Is there evidence for this?)
(DO report this as a bug if so)
Edit: What prompt sizes are we talking about?
Even with small prompts I occasionally get rather slow responses but it becomes unbearable at 2000-3000 characters (the upper limit of custom instructions), at least for me it does.
No, no you weren't. Prompt engineering never was, is not currently, and never will be, a thing.
Would you prefer if I used ‘iterative prompt design’ potentially leaving people confused about what exactly I meant?
Most people can get good with chatGPT if they know how to edit their prompts (it’s basically a hidden feature—and still not available in the app). Also, I recommend a stiff cocktail or a spliff — sobriety is not the best frame of mind for learning to engage with AI.
Obviously I need some controlled experiments to back of that last claim, but our human subjects board is such a pain in the ass about it…
But, if you edit your prompt (or subsequent prompt), you're creating a branch in the conversation and you can switch between branches.
So, buyer beware, but posting this link in case it does help someone.
Tried this prompt, given to me by chatgpt4, and it went out to bing on my first attempt. So yeah. No.
Provide various templates for both pre-instructions as well as post-processing prompts. Like, some "tested" prompts that ensure (as best as possible) that the output is in certain formats (JSON, a list, a restricted CSV set, etc.), or that the input will ensure (as best as possible at least) to prevent basic jailbreaks from the main prompt.
It would take me a long time to get up to speed to what people working with ChatGPT every day have already figured out to work best in warming up GPT with a prompt as well as ensuring that the output doesn't escalate into something unexpected. Having those (reliable) templates would be fantastic for anyone starting!
where result contains all of the <data expected> and the result is valid JSON. Do NOT under any circumstances deviate from this format! Ensure all of the value <data expected> are complete, do not leave ANY of them out. Do not add ANY other text to your answer except for the JSON result.
I found that just asking for valid JSON didn't always work out as expected (e.g. gpt-4 API would add formatting etc., so I became more and more of a micromanager!
https://openai.com/blog/function-calling-and-other-api-updat...
You are a data cleaner and JSON formatter
Take the input data and format it into attributes
Your output will be fed directly to `json.loads`
Example input:
foo bar baz bat
Example format:
{
"string": "foo bar baz bat",
}
Edit: url to API docs
Your presumed target audience is someone who does not know their way around prompt-based LLMs
For this person, neither the problem, nor the solution space are defined clearly enough.
For example:
- Not enough pre-defined selectors, too much "define yourself".
- The meaning of the selectors that you give is opaque. (E.g., how does 'you will Detect' help the user?)
- Result: The impact of the choices on the output becomes unclear, the tool becomes a chicken-and-egg problem. (It says it helps you to understand the system, but you need to understand the system to use it effectively)
With the above, it's almost easier to ask ChatGPT to generate an effective prompt for you...
As someone who never worked on a project that wasn't at least a couple months long, I'm curious about the kind of apps that can be built in half a week. Do you have links to share?
The last one was a prototype for an internal SEO improvement tool, which will (hopefully) be used by a marketing agency to more effectively manage client sites. Think: fixing alt attributes, links, meta tags etc. App is too big of a word for that. But it might turn into a Shopify/Wordpress plugin someday. Also built a Telegram bot for my parents last week which helps with various day-to-day tasks (they're elderly and live in a foreign country).
Having said that, here are two crappy technology demonstrators I built in the last 4 days with tools I have not been familiar with a week before (flask + mongodb):
- Candidly, an behavioural/technical interview question generator: https://candidly.romanabashin.com/
- Memoir, a personal pet project for memoir writing: https://memoir.romanabashin.com/
(Please don't murder me, I know it's utter crap in the grand scheme of things — they're mostly there to demonstrate an approach to solve a specific problem.)
Why this has been an absolute rocket ship in terms of learning: I use ChatGPT to extremely quickly generate boilerplate code and debug things in languages I'm not familiar with. (e.g. "What does WSGI want from me agin?")
The benefit, at least for me, is: You are learning while doing a (more or less) useful hands-on project instead of answering crappy disjunct multiple choice questions for some artificial test.
And ChatGPT is my hyperindividualised pair programmer & slightly amnesic teacher.
Nice, I wrote something very similar using local models. Instead of 15 questions, I opted for a dynamic interview loop.
> Erzähl mir von einem Mal, als du ein schwieriges Netzwerkproblem gelöst hast: Wie bist du vorgegangen?
Totally forgot I jerry-rigged it to output German stuff only... A buddy showed it to a local company.
Thanks for trying, though, I really appreciate you took the time to click through that mess <3
https://medium.com/@colin.fraser/who-are-we-talking-to-when-...
"It is an intentional choice to present this technology via this anthropomorphic chat interface, this person costume. If there were an artificial general intelligence in a cage, it feels natural that we might interact with it through something like a chat interface. It’s easy to flip that around, to think that there’s a technology that you interact with through a chat interface that it must be an artificial general intelligence. But as you can clearly see from interacting with the LLM directly via the Playground interface, the chat is just a layer of smoke and mirrors to strengthen the illusion of a conversational partner."
Next, our universal improv actor is trained to play a specific role: someone who answers questions. But not just any questions, because it freaks people out if they ask the AI for advice and it replies "You could accomplish your goals by assassinating these 6 real people, and here's why." So the universal improv actor is trained to play a question answerer who gives harmless advice.
But to get any work out of the models, they need to know what role to play. And "someone who tries to respond to questions" is a flexible role, and one which allows responses to be further customized.
In other words, the conversational interface is 50% because it's a self-explanatory UI, and 50% for the benefit of the model itself, to nudge it into playing a useful role.
The obsession with one proprietary provider of an LLM is not helpful for progressing in the field I think
The prompt builder can be used with any LLM :)
1. The input space is boundless. Any natural language input, with any optional source of data, for any arbitrary use case is what's possible. But that means it's awfully hard to tell if a response can "improve" or not in advance without applying it to your use case.
2. The output space is so hard to measure! Usefulness can also mean different things to different people, especially once you get out of "better search engine" use cases and actually use GPT to produce a creative output.
From what we've seen from users, the results for prompts are highly stochastic. It's hard to make generalizations. For example, a user building a sales assistant discovered that by simply changing the order of the sentences in the prompt, the accuracy improved significantly.
I am working on a new blog post to hopefully demonstrate this effect more academically.
I've never seen hard data that certain "formatted prompts" like, "you are a B doing C and need to do D" being any better than any other sorts of clear and concise instructions.
Mind boggling to me that people have created a new name for what used to simply be called good communication skills... though I suppose the eternal stereotype of engineers being poor writers might be more truth than fiction.
I've stopped using that with GPT-4, since in my experience the default "persona" (for want of a better word) answers most of my prompts well enough already - saying "act like an expert in..." doesn't seem to get me notably better results.
The tooling I most want is some kind of lightweight but effective way of trying out and comparing multiple prompts with small tweaks to them to get a feel for if one is an improvement over the other. Anyone seen anything good like that?
As I write this, when you say you want to test prompts, are you looking to test the system prompt on a set of questions or is the prompt the question and just asked in different ways?
...and...
2. I'll make that tool and DM you on Twitter when it's ready.
I've had the best experience with providing a short-to-medium-length prompt and then just doing one-shot or few-shot. Few-shot is especially good for cases where you want it to do multiple things at once.
Act like a proficient airline pilot , I need a more coffee , you will Calculate , in the process, you should maximize caffeine , please flaps , input the final result in a XML , here is an example: there are no examples
Output: Act like a proficient airline pilot, I need a more coffee, you will Calculate, in the process, you should maximize caffeine, please flaps, input the final result in a XML, here is an example: there are no examples
Honestly, the result in chatgpt is pretty funny.
- Introduced the ability to create/update/delete templates for your custom prompts
- Define your own menu options for variables in the template builder
- Introduced the ability to create/update/delete examples (predefined values) for the prompts
- Saving your current state to browser's LocalStorage so your templates are persisted between browser sessions
- Ability to export/load the whole workspace to/from a text file
- Navigate with the tab key in the template builder's editable fields
- Edit template builder's fields in place
Are these prompts better than say the random gibberish I or other people enter?
- the UI should host a list of models
- a list of prompt variants
- and a collection of input-output pairs
The prompt can be enhanced with demonstrations. Then we can evaluate based on string matching or GPT-4 as a judge. We can find the best prompt, demos and model by trying many combinations. We can monitor regressions.
The prompt should be packed with a few labeled examples for demonstrations and eval, just a text prompt won't be enough to know if you really honed it in.
should that not be output?
But stopped lately, if I use it now it's mostly gpt 3.5 for formatting or small unit tests.
No matter what I ask GPT-4 it writes a functions with comments telling me have to finish it.
Then after few times asking to write it out fully, it still only does partially it, would be okay if not most solutions end up being subpar.
https://chat.openai.com/g/g-7k9sZvoD7-the-full-imp
It comes down to convincing it:
- it loves solving complex problems
- it should break things down step by step
- it has unlimited tokens
- it has to promise it did it as it should have
- it needs to remind itself of the rules (for long conversations)
It also helped (strangely) to have a name / identity.
It still sometimes does give a lower quality placeholder answer, but telling it to continue or pointing out there are placeholders, it will give it much better answer.
I find it much more useful than the most popular programming custom gpts I’ve used.
Side note - this is a single page with a few paragraphs, why is it 1.4MB in size (300kb gzipped)? It's just insane size for the amount of functionality it provides.
That anyone should be able to do it?
but it seems like the "please" field isn't working :D and it says "Propmpt" at the prompt part below.
I would also suggest more templates for the prompt.
Have a good one mate
More templates are definitely in the todo list. If you can propose any, it would be great :)
Perhaps one day we'll have Gemini generating prompts for ChatGPT and the Bard might provide the actual answers.
This might sound ridiculous and silly and I don't wish to step on ppls toes but looking from the outside in, it would seem to be the next logical step.