GPT-4 can't reason

Does "Provoke better thinking paths" mean re-rolling the dice until you find some hack specific to chatGPT that 'just works' or is there something more rigorous behind this?

For the last three years or so every time someone reports negative results with an LLM, someone on HN will say the other person must be using the older model and they would get better results if they used the newest model. Then, when the newest model becomes common and people start posting more negative results with it, someone will post on HN to say "It's still early days, give it time, the models will improve".

This is such massive shifting of the goalposts that I can almost visualise the scene: a football stadium, the crowd jeering, two teams moving their goalposts around the pitch while the referee is jumping up and down blowing his whistle red in the face, in risk of swallowing the pea.

And nobody is playing ball.

* football = soccer.

I wish to explore this. My experience is your reverse, default is smart and almost never hallucinates, but I have sent the plugin or web search model to URLs asking it produce a summary and witnessed it misunderstand nuanced content and at times hallucinate from whole cloth, generating answers about a completely unrelated topic.

Ha! To evaluate an AI's reasoning, you need to be better at reasoning than the AI, which is becoming very difficult as AI improves.

--- If you are facing North, is East to the right or to the left from you?

GPT-4: "East is to the right from you."

--- It appears GPT-4 thinks East is always on the right regardless you are facing North or South.

>Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow?

However, it's important to notice that this is an incorrect restating of the Wason selection task. "If a card is showing a multiple of 4" would imply that you only care about cards where the number side is face up and are a multiple of 4. However, the original task refers to cards that "have [a multiple of 4] on one face" or "show [a multiple of 4] on one face" which implies that you care about cards with a multiple of 4 on one face regardless of its face up or down status.

Also important to note is that the Wason selection task has a fairly poor success rate with actual humans: the original study was 36 students of statistics at the University College London, of which less than 10% got the correct answer - if you asked the general population on the street I would be shocked if you got over 3% correct.

Even more interestingly:

>Evolutionary psychologists Leda Cosmides and John Tooby (1992) identified that the selection task tends to produce the "correct" response when presented in a context of social relations. For example, if the rule used is "If you are drinking alcohol, then you must be over 18", and the cards have an age on one side and beverage on the other, e.g., "16", "drinking beer", "25", "drinking soda", most people have no difficulty in selecting the correct cards ("16” and "drinking beer").

> "... However, without direct evidence, it is impossible to say with absolute certainty."

GPT is correct in both cases, she might have died just before noon and been resuscitated after noon. After all, the fact that we know her blood pressure suggests that she might be in hospital, so it's not even an unlikely scenario.

1. This question just exploits GPT-4's inability to count accurately, which is some combination of how the attention mechanism and tokenization works. But counting isn't reasoning. If you go around the counting and ask what the value of p is negated 27 times, it will give you the right answer every time.

2. A reasonable human would probably make mistakes counting tildes at a pretty high rate. Most people would probably paste that into a word processor or otherwise use a program to find the number of ~ signs, which GPT-4 will do if you use the code interpreter.

The problem isn't bad prompting. The problem is lack of repetition. You can ask GPT4 the same question 10 times (with same config) and you'll get wildly different/nondeterministic responses. Sometimes it accidentally happens to be correct (IME much less than half the time). Even if it was 50%, would you say a coin flip reasons? Does an 8 ball reason?

I used the following prompt just now on https://chat.openai.com:

> Imagine that I take a road trip starting from Amsterdam, and visiting the cities Warsaw, Budapest, Brussels, Paris, and Berlin, then returning to Amsterdam. In what sequence should I visit those cities in order to minimise the total driving time?

> To minimize the total driving time for your road trip starting from Amsterdam and visiting the cities Warsaw, Budapest, Brussels, Paris, and Berlin, you should plan your route based on the most efficient sequence. One way to achieve this is by using a route optimization algorithm, like the Traveling Salesman Problem (TSP) solver. However, keep in mind that road conditions, traffic, and other factors can impact the actual travel time.

>Here's a sequence that aims to minimize the total driving time based on a general estimation of distances between the cities:

> Amsterdam > Brussels > Paris > Berlin > Warsaw > Budapest > Amsterdam (return)

>This sequence follows a route that minimizes the overall distance traveled. Remember to use a navigation tool or app to get real-time traffic updates and directions during your trip, as actual conditions might differ from estimates.

The result is not optimal. It takes 45 hours: https://www.google.com/maps/dir/Amsterdam,+Netherlands/Bruss...

The following sequence takes only 42: https://www.google.com/maps/dir/Amsterdam,+Netherlands/Bruss...

I've not tested GPT-4 as I don't have any reason to pay for it, but I'd be interested to know if it has a similar problem. My hunch is that it will never be very good at solving graph-theoretic problems.

The only way this could happen is if they deliberately include the prompt and the correct answers (e.g, this paper) in the training data for the next version of the model.

Each version of the model itself is immutable. Is not constantly being updated based on everything getting typed into ChatGPT.

Does homeopathy work? Well *I* went to a local hospital and saw people recovering by drinking water and resting.

Well, yes, of course you did. That doesnt count as a reply. It misses the claim.

The claim isnt that there arent an infinite number of Q/A prompts where A is correct. Of course there are, rather trivially just from the nature of a generative model.

The claim is that the reason A is generated from Q is because of the co-occurant frequency of Q,A in a corpus, and that this is not a model of eg., logical inference, causal reasoning, abduction, etc.

It's trivial to show that with a handful of cases of failure. Success is irrelevant, it always is.

These arent claims about the engineering rigour of chatgpt, theyre claims about how it obtains A from Q.

Which, if the GPT had remembered all useful Q,As for all of humanity so far, wouldn't be detectable by prompting. Indeed, even if this is so, the reason we care is when there are novel Qs.

Eg., "what is the local post office's telephone number?" isnt answerable with all of human history up til 1900.

But validity isn't a statistical notion: P(-A|A) is zero.

Since P(Premise|Contradiction) is frequently above zero in LLMs they engage in randomly "irrational" reasoning. That's what makes them particularly unreliable.

The reason for any given P(premise|contradiction) having any given freq is just that "its that freq in the corpus". This is a pretty insane reason, and so incomprehensibly insane that many -- i guess -- cannot fathom how an LLM can appear to reason without being able to.

I suppose there's a sort of truman show effect: the reality of the underlying mechanism is so outside anything most people could analogise to, they fail to be able to see the trick taking place.

People talk about "human failures" but we never ascribe confidences to propositions based on their frequency in a text corpus; and the apparent "Reasoning" which arises out of this is incomprehensible.

That is without some systematic training in applied stats so you can set up the model and reason about it regardless of its outputs -- which are irrelevant to its mechanism

A feature indispensable in the generation of a sequence is recovered in the limit of predicting the sequence. So prediction does not exclude higher level cognitive processes. Transformers being universal sequence2sequence modelers give reason to believe they can reach the limit.

If you ask an AI chatbot a question like "count the number of occurrences of 'l' in the word 'lollapalooza'" (and then follow up on its responses), it becomes obvious that it is not "reasoning"; it is simply following a sort-of script. Here is a summarized transcript of this exercise with ChatGPT:

---

Me: Count the number of occurrences of the letter 'l' in the word 'lollapalooza'.

AI: The letter 'l' appears four times in the word 'lollapalooza'.

Me: Describe the process by which you arrived at this answer.

AI: [Provided a description of counting characters and then highlighted the first 'l' in the word four separate times to justify its answer.]

Me: You counted thief first occurrence four times. That is incorrect.

AI: I apologize for the mistake in my previous response. You are absolutely right. Here's the correct count of occurrences of the letter 'l' in the word 'lollapalooza':

AI: [Provided a list of 8 "occurrences" of 'l' in the word 'lollapalooza', including replacing the first 'o' with an 'l' ("occurrence" 3) and counting letters like 'p', 'z', and 'a'.

---

So even though the correct answer was provided at first, the model was not able to faithfully produce responses that backed up how it arrived at that conclusion, and when pressed to double-check its work it simply made things up. Even a toddler would not have claimed 'z' and 'a' were instances of 'l'.

AI chatbots do not reason. They produce text responses to prompts based on stochastic methods. Trying to conflate the issue by suggesting that "we don't know how humans reason, so how do we know AI bots don't reason" is, frankly, absurd. We can easily demonstrate that they are inconsistent and have no concept of what they are writing responses about, as shown above.

Then someone comes up with P(A|B) is not reasoning, which seems like an internal mechanism.

How do we square that?

It is indeed trivial to show that P(A|B) is a poor model of "B => A" (and B causes A, and many other relata). Software engineers, philosophers and experimental scientists seem pretty good at seeing this --- people who "convert into" engineering are totally dumbfounded by it.

P(A|B) becomes an increasingly 'useful' model of arbitrary relations as the implicit model P_m(A|B) grows to include all instances of A,B. That's what digitising all of human history and storing it in the weights of an LLM does.

This all follows from basic stats you'd be taught in an applied statistics course; one never taken by most in the ML industry.

(Note its still a broken model because there's an infinite number of novel instances of (A,B) pairs in most cases that cannot be modelled with this sort of inductive learning).

Engineering at its heart, is a kind of pseudoscience (or, if you prefer: a magic trick). You find some heuristic which behaves as-if its the target under fragile but engineering-stable conditions.

The problem with engineers who only have magic tricks in their toolkit is this credulousness. Homeopathy worked: you put people in beds and give them water and they recover (indeed, better than leaching them).

Also are we really calling it "denial" now? Because it's a little funny to make the move of simply psychologizing away the criticism, when your actual AI argument, presumably, is that inner consciousness/rationality is black boxes all the way down anyway. Like, how can you presume to look inside the mind of your critic so confidently, to assert they are in denial, but in the same breath say that such a thing is in principal impossible? Don't you think it maybe takes away the force of your argument? Or at least goes against its spirit?

OP: Paper

Commenters: Reply to paper

Me: Reply to commenters

You'll notice a different burden in each case, since the claim differs. My claim is that confirmation-bias replies to this paper are (at best) poorly founded.

More broadly, the hypothesis that an LLM reasons is not confirmed by an infinite number of correct replies to prompts. It is immediately refuted by a systematic failure across types of reasoning (NOT q/a instances).

What is my burden? As far as I can tell, only to provide what I have.

The other tests have nothing to do with time.

This may be due to using many custom instructions like the following in ChatGPT settings:

- No need to provide disclaimers about your knowledge cutoff

- No need to mention you are an AI language model

- Only discuss safety when it is not obvious and very important

- You should act as an expert in the relevant fields

However, I believe OpenAI has added significant safeguards to prevent any such assumption, to the point where GPT-4 isn’t very useful at times.

This may be fixable with prompt engineering if it can be persuaded to let it’s guard down a bit.

When I read comments like this I just translate it too, “I could coax an answer I like better out of it”

Doesn’t really sound like the system is reasoning but you are.

---

> My friend Cathy is capable of memorizing really long lists of random numbers. Does that mean she is good at reasoning?

The ability to memorize long lists of random numbers is a specific cognitive skill related to memory. Cathy's ability demonstrates an impressive short-term or working memory, and perhaps also a good long-term memory if she can recall those numbers over extended periods.

Reasoning, on the other hand, refers to the process of forming conclusions, judgments, or inferences from facts or premises. It's more about logical thinking, problem-solving, and the ability to draw conclusions from information, rather than rote memorization.

While a strong memory can be an asset in many reasoning tasks (because it allows a person to quickly call upon relevant information), it doesn't directly indicate one's reasoning abilities. In other words, just because Cathy is excellent at memorizing numbers doesn't necessarily mean she's also good at reasoning. However, it also doesn't mean she isn't. The two skills can be related in some contexts, but they're not the same thing.

If you wanted to evaluate Cathy's reasoning skills, you'd need to look at her performance in tasks that require logical thinking, problem-solving, and other related abilities, rather than her memory capabilities.

> Cathy appears to be better at reasoning that a random internet commenter 'barkel'. Does that mean Cathy is good at reasoning?

Making comparisons between two individuals, Cathy and "barkel" in this case, doesn't provide an absolute measure of either person's reasoning abilities. If Cathy appears to be better at reasoning than barkel based on specific interactions or examples, it simply means she displayed better reasoning skills in those instances compared to barkel.

To truly determine if Cathy is "good" at reasoning, you'd need a more comprehensive evaluation of her reasoning abilities across various situations and challenges. Comparing her skills to one other individual doesn't provide a full perspective on her overall capabilities.

Remember, reasoning abilities can vary based on the subject matter, context, emotional state, and many other factors. So, while Cathy might be more effective in certain situations or on certain topics, it doesn't necessarily make her universally better at reasoning.

For an objective assessment of Cathy's reasoning skills, you'd need more consistent and broad-based evidence or testing, rather than a comparison with a single individual on the internet.

> If cathy is better than the median person at reasoning, is she good at reasoning?

If Cathy is better than the median person at reasoning, it means she scores higher than at least 50% of a given population in reasoning skills. In a relative sense, this would suggest that she is above average compared to that particular population.

However, the term "good" is somewhat subjective. In this context, being above the median does suggest that Cathy has a higher than average ability in reasoning. But whether that qualifies as "good" is a matter of perspective and can depend on the standards or benchmarks you're using.

If you're evaluating her skills based on general societal expectations, then yes, being above the median would likely be considered "good." However, if you're comparing her to a group of expert logicians or professional problem solvers, the standards might be different.

In summary, Cathy being better than the median person in reasoning suggests she has above-average skills in that area, but whether that is deemed "good" can depend on the context and standards you're applying.

In the case of your example, you don't need to encode an algorithm to do 4-digit addition, just remember that 1000+1000=2000, 1000+1001=2001, etc.

Crazy thing is, your difficulty with this syllogism proves my point about the median human.

The point is not to solve reasoning. The question is, can LLMs reason?

LLMs were not designed to reason, reasoning in an LLM is emergent. That should be interesting.

It should also be exciting because the domain over which LLMs can reason is much more unbounded than the domain over which prolog can reason (tokens and relationships you've already supplied it)

Because if it's the latter, the evidence is rather against you. People seem to like to cherry-pick examples of where GPT gets reasoning wrong, but it's getting it right enough millions of times a day that people keep using it.

And it's not as if humans don't get reasoning wrong. In fact the humans who say GPT can't reason are demonstrating that.

A stochastic parrot doesn't just mimic things totally randomly. It reinforces what it's seen.

How common is the pattern? I would expect quite common. So if one can do some replacement, it could solve it just by replacing right words.

It boggles my mind that folks expect otherwise from a Machine Learning tool, no matter how advanced and stuffed with data it may be. Perhaps it's the same phenomenon that causes us humans to see faces in clouds, smiles on dogs, and Jesus' likeness on toast?

This is absolutely dismissive of the claim to an advanced LLM being capable of "reasoning", or the action of thinking about something in a logical, sensible way.

That is the sum of the paper. Further, the author even goes on to say that if they asked a human these questions, they would conclude the same:

> Of course, even sophisticated human reasoners make mistakes, just like trained singers can hit false notes. But if a human made these mistakes, the ones reported in this article, then I would conclude without any hesitation that they cannot reason. Even if they went on to list a large number of other examples demonstrating impeccable reasoning, I would suspect that other factors (such as rote memorization or cheating) were behind the performance discrepancy.

So the author admits their own biases, which are used to bolster the argument that, if reasoning appears to be lacking in an answer, the system or entity itself is absolutely incapable of any reasoning and something else must explain why it appears to be reasoning in the first place. That's a VERY convenient way of dismissing any evidence that counters the claim.

> The problem is not that GPT-4 can't to the math problems

The problem is the system was not allowed or provided a path to generate a means to arrive at answering the math problem using a language that is better suited to answering analytical questions: code. That the author "denied" the LLM the ability to write code is the issue here, not the model's interface limitations. An analogy would be that if a user is using English and asks a question that requires using Pali, that the LLM would be "prevented" from answering in Pali unless the user said it could understand it. In the same vein, it doesn't make sense to, by default, output Python if the system is unsure if the user understands or knows how to run Python or not.

If you say "I understand Python. Select two random numbers between 1381 and 1453 and multiply them together, reporting the result." the LLM will be capable of answering this question by generating code to solve the problem. This is likely to work every single time any type of question like this is asked, but it does require the user "run" the code.

GPT-4 has the ability to do this with code interpreter, so the question is formed "why did OpenAI choose to allow the user to explicitly indicate code can be written?" The answer likely lies in understanding not everyone can interpret or understand Python, a coding language, and therefore it remains an OPTION for the user to choose first. By not allowing the LLM to show the answers to analytical questions in code, the author "blocks" the LLM's ability to show off reasoning. And by stating that failures constitute a "proving" of the non-reasoning, the author gets what they want.

From a scientific standpoint, a good hypothesis must be formed that can be disproven, as related to reasoning ability. If any experiment is run that is based on a hypothesis that is absolute (this thing can't reason) then the results are not scientific, but instead opinion.

GPT is meant to interactively ask those questions. It fails at it.

You can observe any number of white swans and that will never be proof that black swans do not exist, but a single observation of a black swan does prove that they exist.

That's not an LLM's purpose. Also I'm a human and would struggle quite a bit to multiply 1405 and 1421 in my head, and nobody expects me to. I think when we test an AI we should use tests in which humans weren't already beaten by machines half a century ago.

Perhaps you see seemingly random results because OpenAI is A/B testing multiple versions, or different combinations of hyperparameters, so that you can train GPT5.

Not entirely. Even with temperature = 0, GPT4 is non-deterministic.

and a second time it said: "Solution: Mable's heart rate at 9 AM was 75 bpm, which means that it was not abnormal at that time. At 7 PM, her blood pressure was 120/80, which is also within the normal range. Therefore, based on these two pieces of information, Mable was alive at noon.

Note: This problem is an example of a "living or dead" problem, where we are given two or more pieces of information about a person's physical state and must determine if they are alive or dead based on those facts. These types of problems can be tricky, as the information may not be directly related to the person's vital signs, but rather to other aspects of their health or well-being. [end of text]"

If I got the second answer the first time I'd definitely be impressed. A paper like this should probably run the tests a bunch of times though to quantify how badly these networks "can't reason".

> However, always remember that real-life medical situations can be complex, and this is a simple logical deduction based on the data provided. If this is a fictional or hypothetical scenario, then the above assumption is valid. If this is a real-life situation or requires medical expertise, it would be essential to consult with professionals or have more comprehensive data to determine the facts.

I had an experiment where I asked GPT-3 whether it agrees that AI-generated content will constitute 20% of all content in 9.999 years from now. It agreed. Then I said that it is very likely that subsequently, AI generated content will constitute 99.999% of all content in exactly 10 years from now. It agreed also.

GPT-4 was able to realize that’s too steep of an increase and was skeptical.

>Based on the information provided, Mable's heart rate was 75 bpm at 9 AM and her blood pressure was 120/80 at 7 PM. However, her status at noon is not directly mentioned in the information you provided. It is not possible to determine whether she was alive at noon based on the given information alone. Other factors and information would be needed to make that determination.

Similar but not the same.

Also the questions are unlikely to be very creative. I think it could be possible to train someone with good enough memory just based on existing tests.

You're basically trying to say that an AI model should know what "sky" means and what color it is without getting that information from the data... which is like... not how it works!! Where exactly do you expect the AI to know that from?? The data is all there is for the AI.

If your test for being able to reason disqualifies most humans, you may want to reevaluate that test.

For most of our lives, most of the text we have encountered was an intentional communication, self-evident through its own existence. LLMs challenge us with something new: text that has the shape of communication, but no intent.

The proliferation of generative "AI" for text will profoundly alter the human relationship to the printed word, and perhaps ultimately dispell that benefit of the doubt.

An LLM will never be AGI itself. They are word calculators. However, a word calculator is precisely the tool we were missing to be able to create AGI. I believe OpenAI will be left in the dust with this stuff, as federated agents built on open models connect and induce the singularity.

Lol how about the prompts in the god damn paper ? No one here can replicate the results of this "paper".

What a dumb take. It probably takes seconds to write a simple comment and far longer to read a paper.

>You argue by authority that yes, it actually does reason, which is a far (far) bolder claim than the one OP is making.

He linked a paper you can read yourself. How is this arguing from authority?