Comment by dereg | Hacker News Reader

Comment by dereg | Hacker News Reader

Parent Full thread

It's really poor work.

There are two conclusions I took from scanning through this and trying to reproduce a few of the reported failures.

1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model.

2. The author is using ChatGPT's GPT-4, leading him to conflate "GPT-4" with "ChatGPT". While you can consider this a shared failure with OpenAI, due to OpenAI's poor communication, anybody doing serious work evaluating these models would know that the first thing you need to do is use the API and pin the model version. In the author's case, he should have used gpt-4-0314 or gpt-4-0613. What I suspect he did is that he just used ChatGPT's GPT-4, and likely the default model at that. (Nobody should ever use the Default model. It's their most heavily performance optimized model and performs worse on reasoning tasks than the Plugins model, even on within-context-size tasks.)

There are huge problems with that, because OpenAI has done both a ton of fine tuning and performance optimization continuously on the default ChatGPT model over time that its performance has ranged anywhere from "I'm pretty sure this is gpt-3.5" to "whoa, this is damn good" (the latter being mostly the model at launch, which was probably the same as gpt-4-0314).

If the author has been working seriously at evaluating models, specifying the model is the first thing he'd do. Perhaps he should explain his reasoning.

> 1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model.

Does "Provoke better thinking paths" mean re-rolling the dice until you find some hack specific to chatGPT that 'just works' or is there something more rigorous behind this?

Seriously this is the part I dont understand about people parroting "prompt engineering" . Isn't it really just throwing random things at a non deterministic black box and hoping for the best?

I find it's more like that silly experiment where you have to make a sandwich exactly as a kid (or adult) writes the instructions. You _think_ you have a good set of instructions and then you get peanut butter on the outside. So, you revisit the instructions to be clearer about what you want done. That's how I see prompt engineering. In that case, you are simply learning how the model tends to follow instructions and crafting a prompt around that. Not so much random, more purposeful.

That isn’t the model reasoning. That’s you figuring out exactly what parameters you need to use to make the model give the result you want.

It’s Clever Hans on steroids

As always, it's unclear on which side of the keyboard the intelligence lies.

> That isn’t the model reasoning. That’s you figuring out exactly what parameters you need to use to make the model give the result you want.

If its to get the model to present a fixed answer, sure.

If its to get a model to do a better job at solving general classes of problems (such as when what you are optimizing is the built-in prompt in a ReAct/Reflexion implementation, not the prompt for a specific problem), that's, at a minimum, different from Clever Hans, even if its not “reasoning” (which is ill-defined).

Can you come up with a method that will get you a good response every single time? Because if you can't, it's not really engineering.

If someone says they're fine tuning a model (which is changing which layers are activated for a given input) it's generally well tolerated.

If someone says they're tuning a prompt (which is changing which layers are activated for a given input) it's met with extreme skepticism.

At the end of the day ML is probabilistic. You're always throwing random things at a black box and hoping for the best. There are strategies and patterns that work consistently enough (like ReACT) that they carry across many tasks, and there are some that you'll find for your specific task.

And just like any piece of software you define your scope well, test for things within that scope, and monitor for poor outputs.

ELI5 layers? Could someone like me see when I've used one layer as opposed to another, when using ChatGPT?

Technically I'm taking a large liberty saying you're "activating layers", all the layers are affecting the output and you don't pick and choose them

But you can imagine the model like a plinko board: just because the ball passes every peg, doesn't mean every peg changed it's trajectory.

When you fine tune a model, you're trying to change how the pegs are arranged so the ball falls through the board differently.

When you prompt tune you're changing how the ball will fall too. You don't get to change the board, but you can change where the ball starts or have the ball go through the board several more times than normal before the user sees it, etc.

You can't see the ball falling (which layers are doing what), only where it falls, but when you spend long enough building on these models, you do get an intuition for which prompts have an outsized effect on where the ball will land.

> If someone says they're fine tuning a model (which is changing which layers are activated for a given input) it's generally well tolerated.

> If someone says they're tuning a prompt (which is changing which layers are activated for a given input) it's met with extreme skepticism.

There are good reasons for that though. The first is the model-owner tuning so that given inputs yield better outputs (in theory for other users too). The second is relying on the user to diagnose and fix the error. That being the "fix" is a problem if the output is supposed to be useful to people who don't know the answers themselves, or if the model is being touted as "intelligence" with a natural language interface, which is where the scepticism comes in...

I mean, a bugfix, a recommendation not to use the 3rd menu option or a "fork this" button are all valid routes to change the runtime behaviour of a program!

(and yes, I get that the "tuning" might simply be creating the illusion that the model approaches wider usability, and that "fine tuning" might actually have worse side effects. So it's certainly reasonable to argue that when a company defines its models' scope as "advanced reasoning capabilities" the "tuning" might also deserve scepticism, and conversely if it defines its scope more narrowly as something like "code complete" there might be a bit more onus on the user to provide structured, valid inputs)

I'm not sure what this is trying to say.

Neither option implies you own the model or don't: OpenAI owns the model and uses prompt tuning for their website interface, which is why it changes more often than the underlying models themselves. They also let you fine tune their older models, which you don't own.

You also seem to be missing that in this context prompt tuning and fine tuning are both about downstream tasks where the "user" is not you as an individual who's fine tuning and improve prompts, but the people (plural) who are using the now improved outputs.

These aren't the contexts that invite the scepticism though (except when the prompt is revealed after blowing up Sydney-style!)

The "NN provided incorrect answer to simple puzzle; experts defend the proposition the model has excellent high-level reasoning ability by arguing user is 'not good at prompting'" context is, which (amid more legitimate gripes about whether the right model is being used) is what is happening in this thread.

That’s science :). We have now established many strategies for context-independent effective prompting via typical experimental research - see https://help.openai.com/en/articles/6654000-best-practices-f...

That is explicitly NOT science.

Almost, but it's more like throwing ideas with an expected outcome at a non deterministic black box and hoping for the best.

No, its not. While GPT-4 (like some but not all other LLMs) is somewhat nondeterministic (even at zero temperature), that doesn’t mean there aren’t things that have predictable effects on the distribution of behavior that can be discovered and leveraged.

Obviously not. For example using Chain of Thought will increase the model's performance[1]

[1] https://arxiv.org/abs/2201.11903

It’s not deterministic, but if you set temperature = 0, you’ll start of find reliable techniques for all kinds outcomes.

There are definitely patterns that transfer across different problems, it's not random at all.

There’s even a term of art for making a plan up front and then hitting it with a low-skew latent space match: “Chain of Thought”. Yeah, it’s seen numbered lists before.

And if at first you don’t succeed, anneal the temperature and re-roll until you’ve got something that looks authentic.

Right, but is it actually logically sound and complete, rather than truthy?

In short, if you cannot use this thing without validating every step it's worthless for logic. You might as well solve the problem yourself.

You got me beat: IMHO these things are plenty friggin awesome already and getting cooler all the time. I don't see why there is so much ink (and money) being spilled trying to get them to do things more easily done other ways.

Language models are really good at language tasks: summarization, sentiment analysis, borderline-creepy convincing chatbots, writing pretty good fiction at least in short form, the list goes on and on. At all of the traditional NLP stuff they are just super impressive.

They already represent an HCI revolution with significance something like the iPhone as a lower bound: it's a super big deal.

But while the details are absurdly complicated and the super modern ones represent an engineering achievement up there with anything ever done on a computer, they still fundamentally predict some probability-like metric (typically still via softmax [0]) based on some corpus of tokenized language (typically still via byte-pair [1]).

And when the corpus has a bunch of conversations in it? Great at generating conversations! And when the corpus has some explanations of logical reasoning? Often passably good at looking logical. And when the corpus has short stories, novellas, and novels featuring conversations between humans and science-fiction AIs? Well they can sample from that too.

But imitating William Gibson doesn't make GPT-4 any kind of sentient any more than it makes me a once-in-a-generation science fiction author.

“Real motive problem, with an AI. Not human, see?”

“Well, yeah, obviously.”

“Nope. I mean, it’s not human. And you can’t get a handle on it. Me, I’m not human either, but I respond like one. See?”

“Wait a sec,” Case said. “Are you sentient, or not?”

“Well, it feels like I am, kid, but I’m really just a bunch of ROM. It’s one of them, ah, philosophical questions, I guess...” The ugly laughter sensation rattled down Case’s spine. “But I ain’t likely to write you no poem, if you follow me. Your AI, it just might. But it ain’t no way human.” [2]

[0] https://github.com/facebookresearch/llama/blob/main/llama/ge... [1] https://github.com/facebookresearch/llama/blob/main/llama/to... [2] https://www.litcharts.com/lit/neuromancer/characters/mccoy-d...

Yes. Yes it is.

I'm kind of concerned about the concept of "bad at prompting".

The hypothesis that I'm working off right now is that natural language has structure to it that happens to match some problem spaces. And this makes sense because people will naturally want to talk succinctly and with a convenient flow relative to the problems they encounter the most. Thus jargon is reborn many times over in different domains.

LLMs are encoding this structure.

So a good prompt is one that provides the LLM with additional information about what you expect the answer to be. And bad prompts provide neutral or disinformation.

This isn't to say that being good at prompts is somehow to be disingenuous about the power of LLMs. What is better? To remember much redundant data. Or to remember simply the right sorts of ways to search for the classes of information you are after.

My concern, though, is that the structure of reality doesn't have to match the way that we talk about it. The Novel and the Inexpressible* will tend to yield hallucinations.

[Although, I've had this concern long before I encountered LLMs. My feeling is that there are many people who can only solve problems that match the way they talk about them.]

* - technically, the difficult or unnatural to express, but I couldn't fit that into a single word.

>I'm kind of concerned about the concept of "bad at prompting".

I have met many people in my life that are terrible at asking questions, so it does have some conceptual reality. But this is also why analogy is so powerful for people. It takes the way a person thinks about $A and applies parts of it to $B so they can more easily wrap their mind around it.

Has anyone written a paper about testing and expressing the power of analogy in LLMs?

Some of it is randomness for sure. However the prompts make sense usually intuitively and there's a big deterministic component -- as which prompts work are likely a function of the training data which has a huge overlap between models.

Edit: And even if the exact same prompts don't work on different models, similar prompts often do.

"Tree of Thoughts: Deliberate Problem Solving with Large Language Models" seems better than just re-rolling the dice. https://github.com/princeton-nlp/tree-of-thought-llm

Phrasing a question poorly yields poor answers from humans. Does rephrasing the question mean re rolling dice until you get a form of question they understand?

The questions were released multiple times and GPT was asked for clarification. If it does not understand something, it should ask about the thing it does not understand rather than go forward.

Even when corrected, it tends to produce wrong results repeatedly by insisting on falsehoods or failing to ensure its logic is complete.

No, it’s how you get good answers and reasoning from humans too

Because human languages are not precise.

Human language requests often require some back and forth, to get on the same page.

It is far more efficient to discuss a problem to solve, than try to waterfall it by wasting time trying to be absolutely painfully clear, without any feedback from your problem solver.

Models quickly incorporating feedback is further evidence of complex reasoning.

it's less "keep rolling dice" and more getting a sense of what will work and what won't. The thing I liken it most to is watching a colleague try to google something for 30 mins and then you find it immediately first try because you know how to get google to do what you want.

Similarly, it happens extremely often that when I watch someone else using chatgpt I see what they're trying to do, and know I would have gone about it another way that would have worked.

>> The author is using ChatGPT's GPT-4, leading him to conflate "GPT-4" with "ChatGPT".

For the last three years or so every time someone reports negative results with an LLM, someone on HN will say the other person must be using the older model and they would get better results if they used the newest model. Then, when the newest model becomes common and people start posting more negative results with it, someone will post on HN to say "It's still early days, give it time, the models will improve".

This is such massive shifting of the goalposts that I can almost visualise the scene: a football stadium, the crowd jeering, two teams moving their goalposts around the pitch while the referee is jumping up and down blowing his whistle red in the face, in risk of swallowing the pea.

And nobody is playing ball.

* football = soccer.

Uh, what? Do you have any source that GPT4 on web is not gpt-4-0613? Because I do not think that is true, my guess is you just have different settings for the decoder

GPT-4 in ChatGPT has changed so much and so frequently. I've spent an inordinate time using both, side by side, and it has mostly failed to replicate the performance of their API offerings. While I don't have examples on hand right now, you can also look for yourself. Just look at the network requests in the web inspector. Here's the description of the Default ChatGPT GPT-4 model:

    {
      "slug": "gpt-4",
      "max_tokens": 4095,
      "title": "GPT-4",
      "description": "Our most capable model, great for tasks that require creativity and advanced reasoning.",
      "tags": [
        "gpt4"
      ],
      "capabilities": {},
      "product_features": {}
    }

Note the context size is 4095. Their model has been heavily optimized for speed and, presumably, cost.

I do not see what that description shows.

From the OAI API, gpt-4 seems to be an alias for the most recent model two weeks after it is released. There has not been a release since 0613.

https://platform.openai.com/docs/models/gpt-4

e: from your edit,

"Note the context size is 4095. Their model has been heavily optimized for speed and, presumably, cost."

No, they are restricting context size to make inference on the chat interface cheaper but that does not mean it is a different model.

Keep in mind that the information in the link you provided only applies to the developer models, which are only accessible via API and OpenAI Playground.

OpenAI's developer relations person confirmed it a little whiles back:

> [..] I made the comment elsewhere, but the models are always changing in ChatGPT.

https://twitter.com/OfficialLoganK/status/166447707026357043...

I would guess max_tokens refers to the number of generated tokens, not the size of the context window. That at least is the meaning of max_tokens in the ChatGPT API.

https://platform.openai.com/docs/api-reference/chat/create#c...

Some time ago when people started reporting degradation, Logan.GPT (Developer Relations @OpenAI) clarified that the API models don't change.

This caused a lot of confusion because people thought that was a claim that ChatGPT doesn't change. He then further clarified that "the models are changing all the time in ChatGPT".

https://nitter.net/OfficialLoganK/status/1664476604658069511

>Nobody should ever use the Default model. It's their most heavily performance optimized model and performs worse on reasoning tasks than the Plugins model, even on within-context-size tasks.

I wish to explore this. My experience is your reverse, default is smart and almost never hallucinates, but I have sent the plugin or web search model to URLs asking it produce a summary and witnessed it misunderstand nuanced content and at times hallucinate from whole cloth, generating answers about a completely unrelated topic.

> Perhaps he should explain his reasoning.

Ha! To evaluate an AI's reasoning, you need to be better at reasoning than the AI, which is becoming very difficult as AI improves.

More like you to evaluate an AI's reasoning, you need to be good enough at reasoning yourself to ensure you're actually evaluating the AI you intended to evaluate.

> Ha! To evaluate an AI’s reasoning, you need to be better at reasoning than the AI

No, you don’t.

You do, OTOH, have to have well-defined criteria for what constitutes “reasoning”.

Reply on news.ycombinator.com