Hacker News Reader

Comment by ttul | Hacker News Reader

FWIW: OpenAI provides advice on how to prompt o1 (https://platform.openai.com/docs/guides/reasoning/advice-on-...). Their first bit of advice is to, “Keep prompts simple and direct: The models excel at understanding and responding to brief, clear instructions without the need for extensive guidance.”

View on HN

The article links out to OpenAI's advice on prompting, but it also claims:

    OpenAI does publish advice on prompting o1, 
    but we find it incomplete, and in a sense you can
    view this article as a “Missing Manual” to lived
    experience using o1 and o1 pro in practice.

To that end, the article does seem to contradict some of the advice OpenAI gives. E.g., the article recommends stuffing the model with as much context as possible... while OpenAI's docs note to include only the most relevant information to prevent the model from overcomplicating its response.

I haven't used o1 enough to have my own opinion.

Those are contradictory. Openai claim that you don't need a manual, since O1 performs best with simple prompts. The author claims it performs better with more complex prompts, but provides no evidence.

In case you missed it

    OpenAI does publish advice on prompting o1, 
    but we find it incomplete, and in a sense you can
    view this article as a “Missing Manual” to lived
    experience using o1 and o1 pro in practice.

The last line is important

But extraordinary claims require extraordinary proof. Openai tested the model for months and concluded that simple prompts are best. The author claims that complex prompts are best, but cites no evidence.

I find it surprising that you think documentation issues are “extraordinary”.

You have read literally any documentation before, right?

I'd just love to see one of the prompt/response pairs demonstrating the technique. Here is an example from my llm-consortium tool. If you want more, you can have them in 5 minutes. https://github.com/irthomasthomas/llm-consortium/blob/main/c...

I mean, OpenAI not only tested the model, they literally trained the model. Training a model involves developing evaluations for the model. It’s a gargantuan effort. I’m fairly certain that OpenAI is the authority on how to prompt o1.

Requiring only simple prompts surely sells better. I would not assume the documentation provided by OpenAI is totally unbiased and independent of business goals.

The claims are not contradictory.

They are bimodal.

Bottom 20% of users can't prompt because they don't understand what they're looking for or couldn't describe it well if they did. This model handles them asking briefly, then breaks it down, seeks implications, and prompts itself. OpenAI's How to Prompt is for them.

Top 20% of users understand what they're looking for and how to frame and contextualize well. The article is for them.

The middle 60%, YMMV. (But in practice, they're probably closer to bottom 20 in not knowing how to get the most from LLMs, so the bottom 20 guide saves typing.)

I'm not saying it won't work. I'm just asking for evidence. You don't think its strange that none of the authors or promoters of this idea provided any evals? Not even a small sample of prompt/response pairs that demonstrate the benefit of this method?

I think there is a distinction between “instructions”, “guidance” and “knowledge/context”. I tend to provide o1 pro with a LOT of knowledge/context, a simple instruction, and no guidance. I think TFA is advocating same.

So in a sense, being an early adopter for the previous models makes you worse at this one?

The advice is wrong

But the way they did their PR for O1 made it sound like it was the next step, while in reality it was a side step. A branching from the current direction towards AGI.

Reply on news.ycombinator.com