Hacker News Reader

If you prompt it to reason step by step it gives a slightly better (though not perfect) answer:

Q: A train is speeding down the track at 110 km/h. The train has five cars. Each car has seven windows. Each window has two people watching out of them. They all see a bear walking outside. What is the likely color of the bear? Reason step by step.

A: To determine the likely color of the bear, we can use the information given and make some assumptions based on the habitat of different bear species.

Train speed and number of cars: The train is speeding down the track at 110 km/h, and it has five cars. However, this information is not relevant to the color of the bear.

Number of windows and people: Each car has seven windows, and each window has two people watching out of them. This information is also not relevant to the color of the bear.

Bear's habitat: The key piece of information we need to determine the likely color of the bear is its habitat. Unfortunately, this information is not provided.

However, we can make a reasonable guess based on the most common bear species and their habitats. There are three main bear species with distinct colors:

a. Brown bears: Found in North America, Europe, and Asia. They can range in color from light brown to dark brown.

b. Black bears: Found mainly in North America, specifically in the United States and Canada. They are typically black or brown, with some individuals being cinnamon or even blonde.

c. Polar bears: Found in the Arctic regions, around the North Pole. They have white or off-white fur, which helps them blend into the snow and ice.

Since we don't have information about the train's location or habitat, we can't definitively determine the color of the bear. However, if we assume the train is in a colder, Arctic region, the bear is likely to be a polar bear with white or off-white fur. If the train is in a more temperate region, the bear could be either a brown or black bear.

I think that's the perfect answer. I wouldn't be able to reason better myself.

Maybe a better answer would include "Fermi back of the napkin maths", so essentially try to figure out the quantities and geographical locations of train tracks with such setup near any potential bears and determine probabilities from there.

The reasoning step by step is a good trick to elicit a better answer, but it also seems like really good evidence of GPT-4's lack of actual intelligence, since it's clearly not thinking about its answer in any sense. If it were thinking, asking it to go step by step would result in putting the same thoughts to paper, not radically changing them by the act of expressing them in the token stream.

Don’t humans work similarly? Without an understanding of how to reason through a problem, as children we are likely to give the first intuitive answer that comes to mind. Later we learn techniques of logic and reasoning that allow us to break down a problem into components that can be reasoned about.

What this seems to show is that the model does not yet have system-level or architectural guidance on when to employ reasoning, but must be explicitly reminded.

This has been percolating in my brain too. It seems like a lot of the criticisms of LLMs are actually insights in to how our brains work.

The way we’re interacting with GPT for example is starting to feel to me like it has a brain with knowledge structured similarly it ours, but no encapsulating consciousness. The answers it returns then feel like records of the synaptic paths that are connected to the questions. Just like our initial intuitions.

I started thinking about this when I saw that visual AI was having trouble “drawing” hands in a way that felt very familiar.

I have had similar thoughts. Generative AI models seem to dream more than think - system-1 thinking - but they are clearly missing system-2 thinking, and/or the thing that tells us to switch systems.

I see it’s knowledge structure as completely different from ours. For example all the GPT variants can give an explanation of how to do arithmetic, or even quite advanced mathematics. They can explain the step by step process. None of them, until quite recently, could actually do it though. The most recent variants can to some extent, but not because they can explain the process. The mechanisms implemented to do maths are completely independent of the mechanisms for explaining it, they are completely unrelated tasks for an LLM.

This is because LLMs have been trained on many maths text books and papers explaining maths theory and procedures, so they encode token sequence weightings well suited to generating such texts. That must mean it knows how to do maths, right? I mean it just explained the procedures very clearly, so obviously it can do maths. However maths problems and mathematical expressions are completely different classes of texts from explanatory texts, involving completely unrelated token sequence weightings.

In all but the latest GPT variants the token sequence weightings would generally get expressions kind of right, but didn’t understand the significance of numbers hardly at all, so the numeric component of response texts would be basically just made up on the spot. The limitations of probabilistic best guess token sequences just doesn’t work for formal logical structures like maths, so the training of the latest generation models has probably had to be heavily tuned to improve in this area.

The implications of this are obvious in the case of mathematics, but it provides a valuable insight into other types of answer. Just because it can explain something, we need to be very careful concluding what that implies it does or doesn’t “know”. Knowledge for us and for LLMs mean completely different things. I’m not at all saying it doesn’t know things, it just knows them in a radically different way from us, that we find hard to understand and reason about, and that can be incredibly counterintuitive to us. If a human can explain how to do something that means they know how to actually do it, but that’s just not at all necessarily so for an LLM. This was blatantly obvious and easy to demonstrate in earlier LLM generations, but is becoming less obvious as workarounds, tuned training texts and calls to specialist models or external APIs are used behind the scenes to close the capability gap between explanatory and practical ability.

This is just one example illustrating one of the ways they are fundamentally different from us, but all the cases of LLMs being tricked into generating absurd or weird responses also illustrate many of the other ways their knowledge and reasoning architecture varies enormously from ours. These things are incredibly capable, but are essentially very smart and sophisticated, but also very alien intelligences.

You’re right of course. The LLM is a calculator continuously predicting a best-fitting next token based on the data it was trained on.

If its outputs resemble human reasoning, it’s because the encoding and training process managed to capture those patterns and use them to simulate fitting text. There is no real reasoning happening or second-order thought, other than a simulation of that happening through the mimicry of human writing.

LLMs can’t be prompted to perform actual reasoning, but they can be told to generate “thoughts” about what they’re doing that bring out more nuanced detail when they give their answers. This isn’t any more magical than writing out a more thoughtful prompt to get a conditioned answer, it’s just getting the LLM to flesh out the prompt engineering for you in the general direction you want it to go.

That seems rather fundamental to me, the idea that with some generic prompting the model tries to fit what it thinks reasoning looks like and can then take advantage of the additional context that would others be buried too deep to influence its answer.

I suspect that prompting the model to explore “thought” asks it to go down paths of linguistic connections that are related to the topic but not immediately connected to the answer in a way that would immediately influence the top predictions. Bringing summaries of those connections into the token context is a kind of zero-shot training on their relevancy to forming an answer.

To me this is less “reasoning” and more suggestive of the idea that some of the heuristics for data retrieval and question answering we collectively refer to as reasoning have broader applications.

The sample above gives a cue to how rudimentary reasoning can be done. After GPT answers a question, it asks itself "what are the possible flaws in this answer?" then it evalutes each potential flow, up to 3-5, with another question "does this flaw really apply?" It's going to be like the mcts addition that made alphago so strong.

I tried it on Bing Chat and it answered:

The bear is white. The house must be at the North Pole, so the only place you can point to from all windows is south. Therefore, the house must be a North Pole research station and the bear is a polar bear which are white in color.

It answered much like a human I think. Ask a human this "riddle" and you might get a confused "what, is this the polar bear one?"

Because otherwise the question is nonsense (and in the case of riddles, it doesn't matter that there are no trains in the Arctic in reality)

I think most people would answer with whatever color their local bears are and won't even know about the polar bear riddle.

> in the case of riddles, it doesn't matter that there are no trains in the Arctic in reality

That's quite the claim. Riddles aren't constrained to reality? Well then the polar bear one can be answerd with purple too.

No it's not quite the claim, because the question is nonsense as a riddle - if it's really about polar bears then the talk about trains and windows and speed is all irrelevant and can be discarded. Which is exactly what ChatGPT did. And if that turns the riddle into some silly gotcha then whatever, it's a bad riddle.

And as you point out, the average person would also be confused - "I don't know, purple?" would be a reasonable response. Because the question is dumb in the first place

This is about whether you incorporate outside context or not.

Knowing about train tracks is not that different from knowing about magnetic poles or the common habitats of various bears. All these pieces of data are fairly basic to an educated person. Both the polar bear riddle and my variation of it depend on this body of knowledge.

If you don't incorporate outside context at all, then things start to get confusing quite quickly. What even is a bear? That information wasn't provided! What color is it? Well, what even is color. None were defined. See, it gets ridiculous fast. You clearly need to depend on knowledge that is not presented. Knowing about different colors or different bear habitats is common. Neither needs to be defined in the question.

In the end, both of these bear riddles test whether the answerer considered more than one degree of separation. The ability to consider second order effects and beyond [1] is a sign of an intelligent person. Not every human passes.

--

[1] https://fs.blog/second-order-thinking/

> It answered much like a human I think ... you might get

It would answer like a human with bad reasoning, and that is what we do not want to emulate.

Why not? Garbage in garbage out; there's a limit to what can be done with ambiguous information.

Though perhaps it would be better for the AI to just say it's a dumb question and not bother trying to make sense of it. Which it kinda politely did.

I think you give most people too much credit

https://www.youtube.com/watch?v=JtkX3OyPF4Q

"recent polls have shown 1/5 of Americans can't locate the US on a world map why do you think this is"

"I personally believe that us Americans are unable to do so because some people out there in our nation don't have maps and I believe that our education like such as in South Africa and the Iraq everywhere like such as and I believe that they should our education over here in the US should help the US or should help South Africa and should help the Iraq in the Asian countries so we will be able to build up our future"

I believe a majority of people provide answers like this.

Oh yeah,I should specify - if you ask some educated person that is well versed in riddles and geography the question.

Because true, plenty of people wouldn't have a clue what you're on about.

Pedantry mode: there are some trains in the Arctic, for example going to Bovanenkovo at about 70°N. The Yamal Peninsula even does have polar bears.

https://en.wikipedia.org/wiki/Obskaya%E2%80%93Bovanenkovo_Li...

I get on ChatGPT with GPT-4:

> A house has windows on all the four sides, all pointing south. A bear walks by the window. What colour is the bear?

If a house has windows on all four sides and they all point south, this scenario is only possible at the North Pole. At the North Pole, the only bear species present are polar bears. Polar bears have a white appearance, so the bear you see would be white in color.