Hacker News Reader

Comment by latexr | Hacker News Reader

> if a decent LLM cannot understand it there is a problem with the prompt.

Ah, yes, the “you’re holding it wrong” argument with a dash of “No True Scotsman” so the goalposts can be moved depending on what anyone says is a “decent LLM”.

Well, here’re are a few failures with GPT-3.5, GPT-4, and GPT4-o:

https://news.ycombinator.com/item?id=38304184

https://news.ycombinator.com/item?id=40368446

https://news.ycombinator.com/item?id=40368822

> Imagine someone not knowing chess and explaining it to them. Would they be able to understand it on the first try with your prompt?

Chess? Probably not. Tic-tac-toe? Probably yes. And the latter was what the person you’re responding to used.

View on HN

But people are holding it wrong. All the prompts you sent except the last are super short queries.

For a successful prompt, you introduce yourself, assign a role to the LLM to impersonate, provide background on your query, tell what you want to achieve, provide some examples.

If the LLM still doesn't get it you guide further.

PS: I rewrote your prompt and GPT 3.5 understood it at the first try. See my reply above to your experiment.

You were using it wrong sir.

Your arguments read like satire. “Yes, you see, the way to get a successful response is to be so overly specific that you begin by explaining the universe then giving the answer in full. You essentially have to spend so much time laying out the nature of the desired response that you already have to know the answer yourself. The trick is to spend so much time on it and be so detailed that you’ve wasted more time and energy (figurative and literal) to write your instructions than it would’ve taken you to think of the answer or ask someone else. And of course, we expect every user of LLMs to behave like this.”

> All the prompts you sent except the last are super short queries.

This one is particularly absurd. When I asked it for the first X of Y, the prompt was for the first X (I don’t remember the exact number, let’s say 20) kings of a country. It was as straightforward as you can get. And it replied it couldn’t give me the first 20 because there had only been 30, and it would instead give the first 25.

You’re bending over backwards to be an apologist to something which was clearly wrong.

Well it is a bit like satire. You have to explain the universe for an unspecialized GPT, like you would do to a layman. There are custom gpts that come preloaded with that universe explanation.

In addition, do not ask facts to an LLM. Give a list of let's say 1000 kings of a country and then ask give 20 of those.

If you ask 25 kings of some country, you are testing knowledge not intelligence.

I see LLMs like a speaking rubber duckie. The point where I write a successful point is also the point where I understand the problem.

I can’t believe I’m having to explain this, but the point I’m making isn’t about the content of the list but the numbers.

> like you would do to a layman.

I have never encountered a person so lay that I had to explain that 20 is smaller than 30 and 25.

> The point where I write a successful point is also the point where I understand the problem.

You have demonstrated repeatedly that you don’t know when you have explained a point successfully to an LLM, thus you have no way to evaluate when you have understood a point.

But you seem to firmly believe you did, which could be quite dangerous.

Careful, explain too much and you end up with programming its behaviour, rather than having an intelligent actor learning by itself. Because otherwise one could say a regular computer is intelligent, provided you explain (in code) every single rule of the game.

Reply on news.ycombinator.com