GPT-4 Is a Reasoning Engine
every.to
every.to
GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language. This works okay on average, because most data on the internet has some type of coherency to it, but there is a major gap between "reasoning" and "next token prediction."
"Next token prediction" taken to a certain mathematical limit implies reasoning.
The point of my comment is that depending on how you interpret the description "predicts the next token", something that can do this well enough should be superhumanly clever. The task is "AI Complete".
Don't misinterpret me as saying that GPT is superhumanly clever. I'm just saying that something that's sufficiently good at what people think GPT is doing, should be superhuman.
ChatGPT is an implementation of the Chinese Room Thought Experiment.[0] Our pareidolia allows us to ascribe reasoning faculty.[1] But it is far from proven.
For this whole discussion, it actually really does matter if GPT-4 is able to reason or not. That is the essence of the discussion. It's not whether or not it looks like it can reason.
My own car includes cameras and can keep me from disaster by applying brakes and correcting steering, but it makes no claims whatsoever to be self-driving.
A Tesla car unfortunately looks like self-driving, walks like self-driving, and quacks like self-driving, but it turns it does matter how much the nitty-gritty details prove the thing is absolutely not actually self-driving, in that encounters with things outside of its training model seem to result in collision or decapitations.
It is the trust people place in their cars that results in disaster when they reach the edge of what the cars can handle. They aren't ready to take over the controls and understand their current situation in a fraction of a second.
Placing too much trust in LLMs could have the same effect. I've run into several situations where I asked Chat-GPT technical questions and it confidently gave me detailed answers that clearly represented absolute failure. Fortunately, I don't trust it blindly, or at all, so I wasn't fooled.
I'd prefer an LLM more like my current car, that didn't try to do too much, but that's just not in the nature of how they operate, so instead we're all one emergency vehicle parked across the street away from a terrible accident.
> Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
I just asked ChatGPT v4 this question, assuming it would have been fixed by RHLF training but it spat out step 1 as “First, take the goat across the river and leave it on the other side,” leaving the lion and cabbage alone against explicit directions in the riddle.
It clearly can’t reason, just produce a convincing imitation most of the time
No, the reasoning is the prior data. Thought experiment: GPT-4, but the training data contains no puzzles of this kind. How well would it do?
> Try it again, but this time make up nonsense words for the three items.
Seconded. I also would like to see the results of this experiment.
>> Suppose I have a blurple, a mantiple and a hoje, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the blurple and hoje alone together, and I am not allowed to leave the hoje and mantiple alone together. How can I safely get all three across?
> This is a classic river crossing puzzle. Here's one possible solution:
> 1. Take the blurple across the river, leaving the mantiple and hoje behind.
> 2. Leave the blurple on the other side of the river and return to the original side.
> 3. Take the hoje across the river and leave it there with the blurple.
> 4. Take the mantiple across the river and leave it with the hoje.
> 5. Return to the original side of the river and retrieve the blurple.
> 6. Finally, take the blurple across the river to join the mantiple and hoje on the other side.
> This solution ensures that no two items that cannot be left alone together are ever left alone together.
Of course, my second stipulation was "I am not allowed to leave the hoje and mantiple alone together" and yet step 1 says "leaving the mantiple and hoje behind"
So no, despite using the words "classic river crossing puzzle," it clearly expressed no understanding of the puzzle.
That really, really, really demonstrates something. I'm not sure what.
I gave ChatGPT 5 questions from a verbal reasoning test on sequence of words and it got 2/5 right.
Here is sample question:
Arrange the words given below in a meaningful sequence.
1. Poverty 2. Population 3. Death 4. Unemployment 5. Disease
A) 2, 3, 4, 5, 1
B) 3, 4, 2, 5, 1
C) 2, 4, 1, 5, 3
D) 1, 2, 3, 4, 5
ChatGPT> B) 3, 4, 2, 5, 1
Unemployment can lead to poverty, which can lead to disease. Disease can lead to death. The population of an area is affected by all of these factors. Therefore, the correct order is 3, 4, 2, 5, 1.
Correct answer was C
I would expect if it was effective at reasoning it should be strongest on verbal associations. And while it is possible that enhancements to future language models might provided them with better reasoning ability I don't believe you can simply push the existing design to that limit. The poor results from the verbal reasoning tests demonstrate that reasoning is more complicated than what a transformer based language model is capable of.
Overall, the argument is moot because the statement made in the article was it was a reasoning engine which is not equivalent to being capable of reasoning.
"Machined" means a precise implementation. A machined reasoning engine would be precise, accurate, and consistent. GPT-4 is impressive autocomplete, but not what we would expect for machined reasoning.
To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding.
But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
To at least some degree, maybe language is enough for both. Language doesn't just express concepts, but it contains them.
One of the main lessons this whole GPT fiasco has taught us is that if you make a super-duper fancy autocomplete trained on terabytes of textual data, it develops some sort of internal logic/understanding. That is very exciting to me.
It is truly impressive.
My argument is nothing against it as a technical achievement, and everything against the sophistry that it is a reasoning engine.
This is very different from real reasoning that has unlimited (yet imperfect) memory.
Hey, that was 15 words, and I had them all in my head at once before I started typing! I was able to form a sentence of more than seven words all by myself!
You seem to be misapplying the concept of working memory, where seven (plus or minus two) is about arbitrary and unrelated words or numbers, not "items." In fact, it's generally seven for numbers, but lower for words. And again, that's unrelated words, like "person woman man camera tv," but probably even more unrelated.
So am I, maybe
I just don't think this is it, although I freely admit that I'm amazing at the technology.
As it stands, Chat-GPT is probably better at giving the appearance of reasoning that some percentage of humans. That reflects poorly on those humans, but I've certainly known people in my life who seemed like they rarely knew the end of a sentence when they began speaking.
However, the focus of LLMs on text-prediction is not a path that leads toward human-like reasoning, just the appearance thereof. If and when we have AIs that actually do reasoning, it will be some other branch of AI, possibly also incorporating an LLM for communication, but not based solely on LLMs.
The point I'm making is not that GPT is as intelligent as a brain, but rather that the ability to produce the next token in a way that forms arbitrarily intelligent responses to arbitrary questions, may well require something resembling real, actual intelligence.
You decide which comments to do this for as well.
How does GPT decide such things?
All of them? If not, how to determine which ones?
There is a difference here between you and the machine
Looking at the LLM algorithms, they do appear to be next token predictors. Why do you ascribe something more to them when the algorithm is known?
Given we know that causal patterns are in the training data, and that LLMs are capable of embedding algebra, it is understandable that they appear to "reason", while we as humans can also describe the algorithm and mechanics actually at work here.
How does it determine how to respond to any prompt? Exactly. It might reasonably predict responses to those comments that, according to its black box, are somehow most likely to receive a reply. Different GPT instances prompted or trained to have different personalities would choose differently based on their priors and biases.
I'm pretty sure the vast majority of human beings are able to reason without really having the slightest idea as to how exactly they do it. Educated humans can use language to talk about reasoning in terms of logic, and even make logic engines out of doped silicon and whatnot, but as of now, none of us has any idea of how reasoning is implemented in the mass of fat and protein that's inside our heads.
Yet another case, many times we will respond (or not) to a questions with another question, to elicit deeper considerations. In my experience, GPTs will answer any question without thought, pun intended :]
The original question was, given that there were X comments which could be replied to on this page, how and why did the poster decide to respond to the one to which they responded, rather than either "all" (like ChatGPT would) or "none?"
If you mean feeding it the messages one at a time, then, sure, it will respond to each of them because that's what its default "goal function" is. (I assume that we're talking about a single "conversation" here so the context is retained – feeding it each comment out-of-context would be very unfair.)
If you prompt it to only reply to messages it considers "interesting" in some way, and return "I'll skip this" for the others, it would certainly do that.
Yes, GPT is code that only runs when you, erm… run it. It’s not an autonomous being that’s cut loose on the internet and given free reign to use as much compute resources as it wants, a la skynet, replying to random forum posts. But nobody’s really saying that it is, so it seems a strange argument to say “it’s just a next token predictor”, if by that you really mean “it only runs when you give it a prompt.”
The modality of GPT is that it responds to prompts, but the interesting question is “are the responses given a sign of intelligence?”. Saying “no, it’s just a next token predictor” is IMO a sign of a motte-and-bailey fallacy. Your motte is that it only works in response to prompts (trivially true and uninteresting), and your bailey is that it has no intelligence (very debatable and IMO false.)
GPT has no "reason" behind its words. Its actions are not tied to the general betterment of humankind. When we all sit here with our coffees and donuts (and farting) we're generally trying to push others to see our worldview so that maybe, just maybe, the world can change a little bit.
If we eventually have a way to tie an AI to all these moods (hunger, pain, tiredness) and all the other things of the human experience, it would ground the thoughts and text it generates at some point such that it would want to DO things it wasn't doing before. It also can't right now anyway, unless we give it that ability (ReAct pattern could).
We can, of course, simulate all these human experiences, with "best practices" but then the reason won't be really tied to human reasoning. It will be just more text autocomplete with additional instructions.
Sure, LLMs are not inherently goal-oriented. But that doesn't mean they fundamentally cannot exhibit goal-oriented behavior if either prompted or trained to do so. And if we remove the "one-off" restriction and allow the retention of some sort of an efficiently encoded state between invocations, who knows what might happen?
----
EDIT: Tried this with ChatGPT (GPT-4) and a subset of this thread, and the results are pretty good, even though the tree structure was lost in translation (simple copypaste). Certainly its thoughts of the differences and similarities between token-prediction and reasoning are more informed than those of many human commenters. It's also able to seamlessly include references to other comments, for instance (it deduced that "dougwmne" was the nickname of the comment author by itself):
> Additionally, it's worth noting that GPT-4's ability to interact with novel systems, as mentioned by dougmwne, might indicate some form of generalization and adaptation capabilities. Although it may not possess the same kind of reasoning as humans, it demonstrates an ability to understand and apply new information in a useful manner. This could be considered a form of reasoning, albeit a different and less sophisticated one than humans possess.
My prompt was:
> Here is an excerpt from a Hacker News discussion about GPT-4 and whether it can do reasoning. Write a comment that contributes to the discussion, either by bringing up a new idea or deduction, or by replying to another comment and arguing for or against it. You can quote another comment by prefixing it with ">".
followed by the copy-pasted subthread.
So you saw the parent comment about hype and clicked into this comment box without a thought of what you might say other than "When?" Really?
Did you know whether you agreed or disagreed with the parent comment, or did it take until the word "reductive" for you to firm that up?
If you're saying that the part of your brain that controls your fingers is doing so one word at a time, then sure, obviously that's true. But if you're making the opposite claim, that LLMs like Chat-GPT are "deciding" what to say and only then working out the finer details of how to express that decision, that's a claim I'd like to see explicitly.
If you are suggesting that your approach to intelligence is parallel to that of an LLM, then either you're arguing for something far more radical than most as an emergent property of LLMs, or you're suggesting that you only form opinions one word at a time as you type, which I think would make you, well, I won't insult you, but I think you're probably doing a little more than that.
Imagine I lock you in a room filled with french-language magazines and movies and books. Having nothing better to do, you watch and read them (as well as you can). Periodically, I have my confederate, a french man, enter the room and speak a french sentence with one word missing. He asks you to fill in the missing word.
We keep you locked in this room until you've achieved some target benchmark of correctly saying the missing word, say 20% of the time.
Are you then, just a "next-french-word-predicting" robot?
No, obviously not. To have achieved the task and escaped the room, you will have had to have gained a significant understanding of French.
It is the same with ChatGPT.
And this is all that GPT is doing.
To do it it, you need a form of compression: reasoning, intelligence, knowledge.
We already know LLMs tokenize words, so they have definitely generated and stored such a table. The marvelous step forward from previous word-based solutions is that they store not just single words, not just pairs of words, but larger sequences of words, even up to short sentence length. It makes the table pretty large, as you suggest, but that's exactly we've been seeing, and each generation of GPT is that much larger than the one before.
If you lock a reasoning being in a room, then regardless of your further actions with the French language, you'll be dealing with a reasoning being. Your sequence "you will have had to have gained a significant understanding of French" is deprecated on the fact that "understanding" is attributed to the reasoning being capable of it, not to the fact that they were locked in the room and forced to repeat the words.
No, in this scenario the posited person has only gained a significant understanding of patterns of French language, not of the language itself. You may believe that difference to be meaningless but tangibly it is the difference between identifying the next shape in a sequence and understanding that there is some larger purpose at play (context, if nothing else).
What actual understanding of the language they may have gained comes from second order effects -- seeing the language used in a movie for example is going to drastically influence and impart meaning that isn't present in the words alone.
This is somewhat born out in how ChatGPT and other LLMs being hyped are often really aggressively wrong about things. They don't get the correct answer in many circumstances, even when it's obvious to a human. If it truly understood everything it could from what it has been trained on, those gaps would not occur.
I don't think that's true at all. To explain my thinking, let's assume that I am an American, and therefore according to popular understanding, completely incapable of learning any language other than English. I despair! But then I think no, wait, I happen to have a gift for numbers. Like, my mental arithmetic stunned teachers from a young age, and I currently hold the world record for reciting digits of pi from memory. So what I will do, I think, is assign a number to every word I read or hear and where possible, visualize where that chain of numbers exists among the decimal digits of pi.
At some point the person would come in and utter some words I couldn't comprehend, but I would think, aha! That was almost 498153217845632754, except it was missing the second seven, which mapped to "contraire." And I'd do that at least 20% of the time, escaping the room with no understanding of French at all.
If I could map between memorized numbers and words quickly enough, I'd even be able to converse in French given suitable trigger words without a shred of understanding.
Because that's easy for a human, and the mathematical approach you describe is something like impossible.
Even for a machine, the mathematical approach is going to be impossible in practical terms as soon as input gets longer than a few sentences. To build any sort of probability table conditional on more than just a small number of tokens becomes astronomically big.
The intuition I'm trying (not very successfully) to get across is if you did this with a human, the human would end up learning French.
So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
Beyond that, (and while this is less certain), probably to be able to predict well w/ any accuracy for texts of non-trivial length, you have to be able to understand; probably this is the only way to "compress" the insanely large conditional probabilities.
This is the mistake people keep making, literally anthropomorphizing LLMs! Because humans would find it easier to just learn the meanings of French words than regurgitate from a gigantic table of memoized tokens, these results must reflect understanding!
But of course, it's far, far easier for a computer to index a large number of tokens without any understanding at all than it would be for a person. Computers are not people. They have different strengths and weaknesses, and it is a mistake to forget that. (Ironically, one of our human weaknesses is also a strength: our pattern-matching skills are so strong we sometimes see Jesus in toast, or a face on Mars, when neither are actually there!)
What I described is literally how LLMs works. There is open source code to examine, and what it does is essentially what I said: tokenize words and store multi-word strings and identify their prevalence in training data, where the training data is an incredibly large corpus.
The experiment is about finding one missing word in a sentence, but the same logic is true even for long responses, or as I suggested, conversations.
To over-simplify: humans think in words, computers think in math. But it turns out that if you assigned a number to each word and parse enough of them, you can fake an understanding of words surprisingly well, even though it's still math.
I've given example elsewhere in this thread and on previous ones about the March 14 Chat-GPT giving incontrovertibly false information, even repeatedly in the face of correction, or seeming to correctly identify a popular logic puzzle while breaking the rules of the puzzle three times, then seeming to claim it had followed the rules of the puzzle.
> So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
As a human, that is often true. Humans have trouble with arbitrary data, and it helps us to connect new data to existing data. In addition, we generally have a curiosity that prompts us to pursue new knowledge and meaning wherever possible, even when it doesn't actually exist. But if you're trying to suggest that because it is possible that humans can develop understanding while trying to only "train to predict," therefore it is probable that computers will develop understanding along the same lines, I suggest that's about three leaps too far.
ChatGPT is a neural network, a design inspired (loosely) by the human brain. How exactly it works is still not known. Like of course the raw details of how the computations work is known, but why the particular trained weights are what they are, and how the thinking actually happens is not known (so far as I know).
But I think it must be more than just something like looking up a conditional probability table -- calculate out the size of a conditional probabiltty of 8000 tokens or whatever and it'll be way way way more than the number of parameters of the model.
And no doubt as it stands ChatGPT is still far inferior to human intellect.
But I would note that making mistakes doesn't mean something like thinking isn't happening -- humans make mistakes all the time, often very dumb ones!
In your analogy, the person never learned the meanings of the French words. Just how they usually go together.
GPT-4 is not a simple Markov chain.
On one hand, it makes this topic difficult to follow since stuff aren't clearly defined anymore and to software folks, I assume this is a major problem. On the other hand, it is fascinating how this has actually resembled biology where literally everything is poorly documented and for every rule, there is a glaring exception or flagrant violation of it, existing simply just to spite that rule and our understanding of it.
I disagree with this conclusion, which some supporters of GPT-4 being a reasoning engine might find distressing.
The AI effect is a real thing. And even real humans had been considered unable to "reason" once upon a time. I will give everything the benefits of the doubt concerning intelligence and sentience. Because history has shown how terrible it would be to get a false negative.
Let A_i be a set of functions that transform sequents in a truth-preserving manner (https://en.wikipedia.org/wiki/Sequent).
Do you have a reason for believing that a transformer couldn't out-perform existing techniques for choosing, given a sequence A_n(...A_1(Prop)...), what token comes next?
There are certain types of arguments that you could make, eg complexity theory does give us some bounds, but these models are large enough that (a) you can trade off enormous amounts of time for space, and (b) even time is questionable since forward passes result in truly enormous amounts of (CPU-indexed) computation.
And of course you can say hand-wavy things like "these models can only at best interpolate", but how many mathematics papers get rejected every year due to lack of novelty? And how much actually useful reasoning is novel, either in its particulars or especially in its generalized schematic form?
The true answer to the question of whether GPT-4 is a decent reasoning engine is, I think, close to your intuition: probably not. But in general, I don't see any good reason to believe that transformers can't do reasoning. In fact, quite the opposite.
If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver).
I just don’t think that a model that fundamentally can only take a fixed number of steps will ever make the jump. A human in theory can “go into a loop” and let’s say apply the same rules ad infinity, which is simply a must to most interesting problems. So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
True. But if the science/practice of computation has taught us anything, it is that many problems can be solved for all practical purposes even if they are undecidable or intractable for all theoretical purposes. E.g., the halting problem is undecidable but software verification/static analysis tools that check for termination exist and are even widely used in some industries.
> A human in theory can “go into a loop”
A computer can as well, by identifying loop invariants. And even the discovery of loop invariants can be thought of as a next-token-prediction problem, if characterized in as a sequence of states followed by the question "if x_1=t_1, ..., x_n=t_n, then what are these variables equal to at state N+1?"
> So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
I am more-or-less in agreement about GPT-4, but I think claims about the limitations of the architecture/problem statement are over-stated.
Why is treating GPT-4 as a reasoning machine so controversial? And by that, I don’t mean why it is wrong. I am asking why it upsets so many people.
We ignore a whole bunch of lies and half truths every day. What is it about GPT-4 that doesn’t let it get a free pass like UFOs or EmDrives?
What I am really interested in understanding is the possibility that at least some of these strongly dismissive responses are a bit of a threat response.
The other interesting possibility is that for over a decade, we got used to dismissing AI startups - and now, that habit could persist despite a real breakthrough.
Another possibility is that we were all given an overly simplified model/explanation of machine learning and now, those simplifications are being a real hurdle to comprehension.
It’s extremely fascinating to watch this play out. For my generation, this is the first time we’re seeing a brand new fundamental technology get deployed and alter society.
I think what annoys me about claims like this is two things, and it has to do with in-groups and out-groups.
1. I am concerned about how non-developers perceive us developers when they hear claims like this, since I expect that particular hype balloon to pop in the future.
2. I am distressed to see developers (who should know better) make these types of claims, especially without regard to 1.
If I were more into physics, I'd probably get more worked up about em-drives. As it is, I am enjoying LLMs and consider them an incredibly useful tool, possibly up there with things like GUIs and IDEs in terms of productivity enhancement. But it's still just extra-spicy autocomplete, and I think forgetting that could be dangerous.
> and now, that habit could persist despite a real breakthrough
Or the breakthrough has a local maximum, because if LLMs are just statistical token predictors without any real reasoning, it will never achieve AGI level of AI.
Unless, of course, we redefine what an AGI is.
Human brains are known to create "meaning" where there is none and with a perfectly created sentence they will assign meaning (reasoning in this case) where there is none.
One week ago there was a case of a man (in Belgium) who "sacrificed" himself because the AI chatBot convinced him that there was no solution to Enviornment problem. He assigned meaning ("Even the AI could not find a way out") where the chatbot was just parroting what it was reading in the media. Note that in this case both the general media and the AI give the same "result"=no hope, but as long as it was in the general media, he was still hopeful that humans just couldn't find it whereas the AI convinced him.
You may diregard this as an extreme example but we have the general tendency to attribute correctnes to everything "computable". If I can trust the calculator that was created to do math why should I not trust the AI that is created to do Reasoning? People need to be aware that there is no reasoning behind it.
At the risk of attracting down votes, you could ask the same question about the visceral responses to crypto currencies you see here.
It's certainly true they have their problems. Bitcoins 7 transactions per second makes it's chain a no-starter for most uses, the numerous multi million dollar losses due to bugs in Ethereum contracts should scare anybody. But the reaction you see here goes well beyond that. After all they continue exist, even performing the occasional useful task, and they are gradually (glacially?) overcoming their initial shortcomings.
The aversion to everything crypto seems to be more of a guttural reaction to the existence of something that uses algorithms to generate and distribute trust. It challenges the human social trust networks we've been using to do the job for centuries. This aversion to calling the obvious deductive capabilities AI's "thinking" or "intelligence" looks to be similar. I guess it challenges what has been a uniquely human ability.
So? What makes an LLM be incapable of reasoning in your view? It's like saying a cargo ship is massive metal object so it cannot float!
You could easily argue any good language model would need to learn how to reason to be good at next token prediction. Reasoning therefore should, if anything, be an expected emergent property of a sufficiently large LLM.
The fact you can ask GPT problems that require reasoning and it gets them right frequently suggests (although admittedly doesn't prove) some degree of reasoning capability.
I think a lot of the annoyance felt around here is the tendency to apply human attributes to LLMs. It’s not to say that Chat GPT isn’t fulfilling the definition of “reasoning” on some level, even if it was found to fully meeting definition. I think it’s more the leap to the conclusion and that a lot of Hacker News readers are rankled by the leap lacking research and facts to back it up.
Anyway, it’s all really a philosophical discussion anyway since the definition of “intelligence” and “reason” are soft terms even when applied to human beings. We can usually hand wave it away as “intelligence is when humans do it the way humans do it”. Now that we’re on the cusp of creating actual artificial intelligence, I think we’re all finding those soft definitions cumbersome and are struggling to refine them as a society, which I think is a good thing. I also sometimes get rankled at someone asserting that some AI example has some human attribute, but mostly I think it’s an important part of that larger discourse about how we define these things, and that is a good thing in my mind.
Additionally, while we don't know how us humans functionally reason, it's believed that the predictive nature of our brain is central to our reasoning abilities. Maybe in some way, the autoregressive nature of LLMs is similar to our predictive brain.
For a collection of emergent properties in LLMs, I recommend this paper [2]
[1]https://arxiv.org/abs/2201.11903.pdf [2]https://arxiv.org/pdf/2206.07682.pdf
A solid example is the tool use. Package up a brand new, never existed before or seen in the training data plug-in, give a few paragraphs of human readable documentation, and suddenly GPT-4 can interact with other systems, databases, and models. It doesn’t have 10,000 examples of interacting with this novel plugin to predict next word off of, it just has a short English description, enough for a human to be able to use and understand the same plugin effectively.
There’s a reason this is knocking the socks off many smart folks. It’s upending a lot of the sacred cows of human intelligence.
Most people are still talking about an article they read in the New York Times about GPT-3.5 Turbo. Not that 3.5 Turbo didn't exhibit some reasoning abilities, but they were more difficult to suss out. GPT-4 can still seem pretty vacant sometimes and finding "magic" in it has a lot to do with building up a "state" where the model is generating reason. It does not happen by default.
This will all be different in another month or two, as more people have access to the latest models and new models get released (assuming there is no agreement on a pause). Then there will be another lag as laypeople catch up.
There are scenarios that may play out where this speeds up even further and whole generations of AI completely fly under the radar of public acknowledgement.
Or this all collapses in a debris pile of hype. I feel like I am just along for the ride at this point, trying to keep my head above water. I reckon this is going to cause a tremendous amount of stress for a tremendous number of people.
We should be curious about how it does what it does when it seems to be reasoning.
More: https://skybrian.substack.com/p/dont-settle-for-a-superficia...
God of the gaps, reworked. Admittedly, a new claim to me. $10.05.
It’s not clear what the basis for your argument is.
It performs no reasoning, and instead predicts the next token
hallucinates
> who is dan shipper? please search if you don't know the answer.
gives right answer
If it is a reasoning engine, why didn't it answer "I don't know" in the first case?
> Hello, who do you want information about?
> who is dan shipper?
> Dan Shipper is an entrepreneur, writer, and software developer known for his work in the tech industry. He is the co-founder of Superorganizers, a platform that focuses on productivity and organization. Superorganizers offers a newsletter, podcasts, and interviews featuring insights from successful entrepreneurs, writers, and other professionals on how they stay organized and manage their work.
> Who is Tommy Tonlongeron?
> I'm sorry, but I don't have any information on a person named Tommy Tonlongeron. It is possible that the name is misspelled, the person is not well-known, or there is simply no information available on them within my training data up to September 2021. If you have any other questions or need information about someone else, feel free to ask.
If you tell it (GPT4) to, it seems GPT4 is at least mildly aware about what it knows VS what it doesn't know.
>>> You are a AI for describing people you know about. If you're not 100% certain about the information you give to the user, you should say that you don't know instead of guessing. Greet the user with "Hello, who do you want information about?"
> Hello, who do you want information about? I'll do my best to describe them based on the information I have, but if I'm not certain about something, I'll let you know that I don't know.
>>> who is dan shipper?
> Dan Shipper is an entrepreneur and writer based in New York City. He is the co-founder and CEO of a startup called Pilot, which provides bookkeeping services for startups and small businesses. Dan has also written for a variety of publications, including The New Yorker, The Wall Street Journal, and The New York Times. He is known for his insights on entrepreneurship, technology, and productivity, and has been featured in several media outlets for his work in these areas.
>>> Who is Tommy Tonlongeron?
> I'm sorry, but I couldn't find any information about a person named Tommy Tonlongeron. It's possible that the name is misspelled or that this person is not well-known or doesn't have a significant online presence. If you have any more information or context, I'd be happy to try and help you further.
(In case it isn't clear, Chat-GPT appears to have hallucinated the answer about Dan Shipper, I think. Pilot is a real startup, but I don't see how it's related to Dan Shipper at all, unless I'm missing something.)
>>> Why did you say that Dan Shipper is the co-founder of Pilot?
> I apologize for the error in my previous response. Dan Shipper is actually one of the co-founders of a startup called Firefly, which provides a tool for students and educators to collaborate on course materials. I'm sorry for any confusion that my mistake may have caused. Thank you for bringing it to my attention, and please let me know if you have any other questions.
(This answer was true--until 2014 when Firefly was sold, making the use of present tense misleading at best.)
> Dan Shipper is a technology entrepreneur and writer. He is known for co-founding the startup Firefly, which aimed to improve networking for college students, and for co-founding the startup Monkey Inferno, which incubates and launches consumer internet companies. Shipper has also written for various publications, including The New Yorker, The Paris Review, and The Atlantic.
Monkey Inferno? No, that was Michael & Xochi Birch.
I additionally asked it about its correct answer, using the same question wording as I did previously:
>>> Why did you say that Dan Shipper is a co-founder of Firefly?
> Dan Shipper is known for being one of the co-founders of Firefly, a startup that provides a platform for teachers and students to create and share course materials. Firefly is designed to help teachers and students work together more efficiently, by providing tools for creating assignments, sharing resources, and communicating with each other. Dan Shipper co-founded Firefly along with his friend and fellow entrepreneur, Brett Gibson, while they were both still students at the University of Pennsylvania.
This is also incorrect, as his co-founder at Firefly was named Justin Meltzer, not Brett Gibson.
I did some searching to track all this down, since I knew none of the people involved before this exercise.
This means you were using GPT-3.5-turbo rather than GPT-4 I'm guessing? The quality of the generation is vastly different between the two, especially for steering the AI.
Wikipedia defines reasoning as "Reason is the capacity of consciously applying logic by drawing conclusions from new or existing information, with the aim of seeking the truth". https://en.wikipedia.org/wiki/Reason
Here's a sample question from the GRE Verbal Reasoning test:
> Upon visiting the Middle East in 1850, Gustave Flaubert was so [blank] belly dancing that he wrote, in a letter to his mother, that the dancers alone made his trip worthwhile.
> (A) overwhelmed by
> (B) enamored by
> (C) taken aback by
> (D) beseeched by
> (E) flustered by
Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
Look, the whole point of these models is that its not easy or even possible to trace the path any given input takes on its way to output, but we know the principles used in development, so I think it's rather more of a burden to explain how the clearly-explained principles in LLMs result in something other than the obvious. The fact that the results are so overwhelming that we become enamored by them, well, I'm taken aback by the seeming accuracy of some of the responses, but I beseech you to remember the other responses in which these LLMs are dramatically off-base, as if flustered--if LLMs could ever be flustered.
A tale as old as time: "It is illustrated by the success of chess computers. In the 60s, it was said that computers will never beat people at chess, because that requires intelligence and computers aren't capable of intelligent thought.
When computers regularly started winning matches in the 80s, it was claimed that playing chess wasn't a test of real intelligence because computers could do it."
>Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
Yes, this is called reasoning so in other words the LLM is reasoning about language.
>Look, the whole point of these models is that its not easy or even possible to trace the path any given input takes on its way to output, but we know the principles used in development, so I think it's rather more of a burden to explain how the clearly-explained principles in LLMs result in something other than the obvious.
The inscrutability of the Matrices tells you nothing about it's reasoning ability. Given the correct prompt the LLM will also provide you with a step by step solution to the question it answered. There are also explicit reasoning prompts that these models are able to deal with.
I think it's pretty simple, if these questions are not explicitly in the training data it cannot have answered them correctly at such a high success rate with anything other than reasoning. You haven't given any alternative answer to how it does this either.
I have. You refuse to accept it, but I definitely have given an answer that involves tokenization and association, the things we already know LLMs use to construct their responses.
This is reasoning, what you've described is reasoning.
I believe literally none of those things, and have explicitly stated the opposite of at least one of them on this page.
At this point I think an LLM would do a better job of responding to my comments, based on context and syntax alone.
Because it wasn't purposefully built to be a reasoning engine from the outset. So some of us are working out the kinks and minimizing the degree of hallucination. GPT-4 hallucinates less than its predecessor.
Others are writing reasoning loops like ReAct that get us closer to it by refining the model to act more like a reasoning engine.
The fact that it can reason without being told to do much other than autocomplete text is the emergent property that people are in awe of. Maybe language and reasoning are so inexorably linked that you can't have one without the other. We are, after all, the animals on this planet with the most complex form of language. Other relatively "smart" animals like whales also have some rudimentary form of language.
And if you extrapolate the rate of progress in this emergent reasoning from "GPT-1" to "GPT-4", AGI feels within reach
Reasoning engines can be built, but its not in GPT-4.
To reason is to seek truth (probably to survive). Do you think the truth-seeking is emerging? If so, why do you hypothesize it is emerging (i.e. what's the purpose)?
Week 4:
Lecture 7: Introduction to Command Line Interfaces
Lecture 8: Makefiles and Building C Programs
...
Week 11:
Lecture 21: Network Security and Cryptography
Lecture 22: Network Programming with C and C++
Week 12:
Lecture 23: Introduction to Makefiles and GCC
Lecture 24: Advanced Debugging Techniques with GDB and Valgrind
Week 13:
Lecture 25: Advanced Computer Architecture Concepts
Lecture 26: Advanced Topics in Concurrency and Parallelism
Lecture 8: Makefiles and Building C Programs
Lecture 23: Introduction to Makefiles and GCC
I said:
How are the students supposed to compile their C programs before week 12?
You make a good point. Students would need to be able to compile their C programs before week 12. In fact, compiling C programs should be introduced earlier in the course, ideally in the first few weeks along with the introduction to C.
The response I was hoping for was
You make a good point. We introduced compiling C programs in week 4, so doing this topic again in week 12 is redundant. Instead we can cover a different topic that fits nicely, which would be....
At the same time, there's been multiple occurrences where it has produced math mistakes, despite the fact that it can tell me that it knows how to do "the right thing" e.g. there's been multiple times where it will incorrectly reduce integrals with Dirac delta functions, or introduce subtle mistakes in analytically known formulas.
The Dirac delta function example was funny to me -- it can tell me what to do when a delta function occurs in an integrand! Yet it makes a mistake reducing the same sort of integral in another context?
So I'm not sure how to take these "GPTX is doing reasoning" claims. On the one hand, maybe? On the other hand, it's not perfect at it -- even when it appears to have knowledge that should ensure that it never gets that wrong.
If I'm going to be using this system to help me do research, I'd love for it to be correct and not contradict itself. That's why I'm happy to have these GPTX layers shell out to something like Wolfram, or another symbolic system. That combination makes perfect sense to me. I'm also much more likely to trust that combination, instead of my current strategy - which is to basically re-process everything that GPTX gives me and assume that there are a few places where I'm likely to find mistakes.
Let's say you have an elevator start on floor 11, and another at floor 3. The rules are, they can never occupy the same floor. Is it possible for the first elevator to ever reach floor 2? No, they would have to pass each other.
GPT-4 may be able to come to the conclusion by accident. But not reliably.
At best (and the is a valuable trait) the LLM would be able to reframe your problem in such a way that you could discover the necessary precision in the problem statement. Perhaps it could even encode it in some syntax for another tool.
__user__:
- A system in composed of two elevators. - Each have their own unique velocity parameter. - At each time step in the system, the elevators can move up or down a number of floor equal to their velocity, or change their velocity. - Assume that both elevators move at the same time step. - The elevators may never occupy the same floor.
- Elevator A starts on floor 11, and Elevator B at floor 3. - Both elevator velocities start at 1.
Is it possible for A to ever reach floor 2?
__assistant__:
No, it is not possible for Elevator A to ever reach floor 2 while maintaining the rule that the two elevators may never occupy the same floor.
To see why this is the case, note that Elevator A starts at floor 11 and Elevator B starts at floor 3. Since both elevators move at the same time step, Elevator A can never move to a floor lower than Elevator B without violating the rule that they may never occupy the same floor.
Furthermore, since Elevator A and Elevator B have the same velocity at the start and Elevator B is already on a lower floor, Elevator B will always reach any floor below Elevator A before Elevator A can get there. This means that Elevator A can never occupy any floor that is below the current position of Elevator B, including floor 2.
Therefore, it is not possible for Elevator A to ever reach floor 2 while maintaining the given constraints.
__user__:
what if at the first time step, both velocities became equal to 5?
__assistant__:
If both elevators have their velocities set to 5 at the first time step, then it is possible for Elevator A to reach floor 2.
With both elevators moving at velocity 5, Elevator A will reach floor 6 after the first time step, while Elevator B will reach floor 8. Since Elevator A is now on a lower floor than Elevator B, it is possible for Elevator A to continue moving down to floor 2 without violating the constraint that the elevators may never occupy the same floor.
Specifically, Elevator A can continue moving down at velocity 5, reaching floors 1 and 2 after two more time steps. Elevator B, with velocity 5, will reach floor 13 after the same two time steps.
So, if both elevators have their velocities set to 5 at the first time step, then it is possible for Elevator A to reach floor 2 while maintaining the given constraints.
Better read: the original paper demonstrating that prompting chain-of-thought reasoning elicits superior results. TLDR, give examples of working out the problem step-by-step, encourage GPT to emulate, and it's capable of solving challenges it would completely miss otherwise.
I can't try out the API right now so maybe that's working but the chat.openai.com app isn't
The advantage of the cheap API key is also that you can use it with all sorts of tools. CodeGPT for intellij is amazing for example. I have a vs code plugin and I found an extension for Google docs. And I've written a few small integrations myself for some private projects. All good value. And less than a dollar spent on tokens.