Hacker News Reader

Comment by JoshTko | Hacker News Reader

The more I think about it the more I'm convinced I am basically just predicting/saying my next word whenever I speak.

Honestly I feel like one reason people are struggling with this is they can't accept a critical part of the truth : most people are stochastic parrots themselves most of the time. True, creative genuine deep thinking is an exceptional state of thinking for us.

Years ago I transitioned from a developer role to a manager role and suddenly I had to do a lot more talking. Not all the talking needs to be a deeply involved exchange of complex ideas, a lot of it serves a different purpose. Sometimes it can be a simple as filling up the time in a pleasant way with a group of people that may or may not know each other that well.

After getting some experience with this I noticed that I had developed a talking on/off button in my head. I could just simply turn it on and start talking. I could generate words that sounded good together and fit the purpose of the moment. But They just seemed to come from a different place in my brain than my conscious mind. Because that was not involved in this process at all. The only job my mind had was to turn the button off again at the right moment, for the rest it was free to think whatever it wanted.

(I transferred back to development a couple of years later.)

The very fact that we can be conscious about our ability to generate language and work it separately from the rest of our consciousness tells me that chatbots might emulate our language generation ability. We shouldn't invert it to conclude that means they can reason about it.

> We shouldn't invert it to conclude that means they can reason about it.

Agreed. However, I think it's somewhat accepted view that the bulk of work constituting reasoning happens subconsciously, with the conscious mind playing the role of a censor/gatekeeper, and occasionally handholding during when reasoning through tougher problems.

This is why I am starting to think that it is very important for us to be able to "introspect" and reason about the thoughts/feelings we're having. It allows us to have an internal feedback loop that I think a lot of people lack.

Maybe in casual conversation but that's not how I experience my though process about anything non trivial at all. I usually spend a lot of time thinking about the concept in non verbal terms and that process involves recalling images and sensory information in fairly abstract terms and then through what feels like several iterations it starts to coalesce into something I can encode in language.

I think we can all agree that these LLMs are surprisingly good at generating text that is often coherent but I don't see how you can discard all those extra inputs and claim you have the same process.

> thinking about the concept in non verbal ... abstract terms

The abstract terms we think about are concepts, and we think about multiple concepts, at various levels of abstraction, and their relationships to each other, before getting a sense of what we want to say or write.

Only then do we begin speaking or writing, grouping concepts into paragraphs, breaking them down into sentences and words.

And there's evidence that LLMs do something similar, creating embeddings for both big ideas and small details, modeling how the small details combine into larger concepts, discovering the relationships between concepts, and only then generating a probabilistic sequence of tokens to express those deeper concepts.

Right but they're limited to concepts that are derived from linguistic tokens. Human reasoning can include nonverbal inputs.

For example asking GPT what would be a reasonable way to stack a list of arbitrary objects on top of each other.

Lately I've been think that staple of science fiction, the android robot, is the only way to get AGI. I thought a lot about computer vision when that was the bleeding edge of AI a few years back. It strikes me as obviously wrong to try training models on two dimensional arrays of pixels and expect them to develop visual skills. Why don't we give them two cameras and arms so they can learn about physics and start differentiating 3 dimensional input? I think when we have robots capable of interacting with the world as humans do we might be able to train them to develop near human levels of intelligence.

The question is whether you knew how that sentence was going to end when you started writing it, or indeed whether I knew that I was going to add this comma-separated adjunct when I started writing the preceding clause, and I cannot honestly say at this precise moment of typing whether the final word in this sentence is going to end up being 'yes' or 'no'.

I do wonder how well (quickly) the whole thought is formed in your head and it’s just the encoding into language that tricks you into thinking you didn’t know what the word would be.

Having learned another language, the moment you start to feel “fluent” is when you start speaking first in the 2nd language and aren’t using your first language as an intermediate step to translate to your 2nd language.

For me it's a little more subtle. When I sit down to write, say, an email, I usually have a specific point that I want to make but I usually haven't yet figured out exactly how I want to get there. I'm not even sure it is even possible to conceive of writing without doing it word by word. Perhaps the difference though is that we have the ability to go forward and backwards in the sense that we can usually think faster than we can type or write which allows our thoughts to advance beyond the tokens we have produced. This allows for a kind of real time editing process that incorporates both past and possible future information. But I imagine there us probably a way to enhance LLMs to branch and edit before producing the next word materially (if they aren't already doing that).

To go even more meta, there is an analogy I'm trying to make right now in which I am visualizing a road and thinking about how describing the road relates to the process of writing. In my mind's eye, I can see the full length of the road and all of its contours but I can't actually describe the individual stretches of the road coherently without enumerating them. Something similar happens with writing. I can visualize what I want to say far beyond the next word, but it's true that the actual process of writing goes word to word, much like how the process of token selection is described for an LLM. The question is whether the LLM has an analogous conception of where it is going. Going back to the process above, sometimes I know where I am going and haven't yet figured out how to articulate it yet. It is through the process of writing that I am able to articulate that thought. But the thought preceded my articulation of it. I don't know to what extent LLMs have coherent thoughts that they are articulating or if that even makes sense for the type of intelligence they project. My suspicion is that they don't have additional sensory inputs beyond language that give thoughts the immaterial shape that then is expressed in language. Without that, I am skeptical that they will truly get beyond regurgitating and/or remixing what has already been fed to them textually. That doesn't diminish how amazing they are, but I am somewhat more in the Brooks/Knuth camp that they are impressive and surprising, but there is something that ultimately leaves me a bit cold about them.

I usually have a specific point that I want to make but I usually haven't yet figured out exactly how I want to get there. I'm not even sure it is even possible to conceive of writing without doing it word by word. [...] In my mind's eye, I can see the full length of the road and all of its contours but I can't actually describe the individual stretches of the road coherently without enumerating them.

Not to trivialise the interesting point you’re making, but do you never write with an outline? Write bullets for the big points you want to touch, then go back and flesh out the details?

This is a thought that LLMs caused me to have: the emotion of surprise is just the brain model of the world discovering that it had been poorly calibrated, or making poor predictions.

Sounds like the free energy principle [0]:

> The free energy principle is based on the Bayesian idea of the brain as an “inference engine.” Under the free energy principle, systems pursue paths of least surprise, or equivalently, minimize the difference between predictions based on their model of the world and their sense and associated perception.

[0]: https://en.m.wikipedia.org/wiki/Free_energy_principle

Why not? Your brain is already making predictions about what you expect to see and hear as a part of your perception of reality anyways.

Right, but LLMs suggest that learning how to predict and only training on predictions is sufficient to learn anything and to have emergent generative abilities. What if learning to predict the upcoming input is all that's all that is needed for general human intelligence? What if it is all that any animals do?

This right here. I actually am strongly starting to believe that this is indeed what's going on.

I read the book Kingdom of Speech a few years ago and that also left me with the perspective that perhaps language has a lot more to do with how we think and perceive the world than most people like to admit. The book has been heavily criticized but I believe it made an interesting point about language.

I mean hasn't it been proven on multiple occasions that humans who never learn a language during their life also don't develop higher cognitive skills and stay on the level of a child? Clearly language plays a notable part in reasoning, and it's frankly no surprise that we're seeing this behaviour from language models.

It's also quite interesting how foundation models and fine tuning appear analogous to being born with a brain that already has six million years' worth of weights in it (trained implicitly through random changes and natural selection), which are then adapted over the course of a lifetime when environment relevant data is gradually obtained.

The sad truth is that we are all NPCs without free will. "Thoughts" are a non-physical phenomenon that, seemingly, arise from physical activity in the brain. The thoughts (our "experience") come after the electrochemical reaction that manifests them. How could these non-physical phenomena then turn around and influence the physical chemistry? They can't. By the time we experience a thought, the physical state for it has been represented, and that state causes the next one after it. Like a stone bouncing down a hill - chaotic but still deterministic. We're just along for the ride.

The act of being conscious of our own thoughts and using them to influence our physical state is skill practiced by many people with success.

Nope, that's just what it feels like we're doing. It's a good trick!

Reducing a debate that has been raging for millennia across several disciplines to three lines of text tells me you don't have enough of a grasp on the subject to draw any conclusions about it, let alone resolve it altogether.

Care to enlighten us a little? What is there even to debate? The position expressed by your parent comment is so uncomfortable that it's constantly being challenged, but that doesn't make it wrong. Or are "thoughts" some sort of magical fluff not bound by the laws of physics? Are we constantly pulling ourselves by our bootstraps? Giving birth to entirely new brain activity out of nothingness?

This is just woo woo nonsense. Until someone can find an explanation that makes more sense than "natural selection slowly gave apes the ability to think abstractly and they started thinking about themselves", I won't believe in any kind of free will, and now the meta is reaching the point where we create entirely new brains out of silicon.

"They just repeat stuff!"

So do we

Does free will exist or only hard determinism can be true? You seem to have reached a definitive conclusion. And you seem to think natural selection and evolution of species and autonomous machines prove your point.

You also seem to think that questioning the veracity of your position is engaging in magical thinking. How could I possibly 'enlighten' you if you are sure of being right?

It doesn't matter if I point out that physics hasn't been able to prove materialism. Where is the fundamental particle? Why can't we determine even through which slit did a quantum of light travel? The Copenhagen interpretation of quantum physics appears to directly contradict your claims of hard determinism being the only valid explanation of reality. But I must be wrong about that. Since clearly you have seen reality for what it is, and I have failed to do so.

Then again, we cannot escape this situation seeing as it is determined.

Of course I'm willing to change my mind, but so far no one has ever given a compelling argument. I just follow Occam's razor.

> The Copenhagen interpretation of quantum physics appears to directly contradict your claims of hard determinism being the only valid explanation of reality

OK, fundamental physics might not follow hard determinism (or not in a way that we currently understand), but please, indeterminism says nothing about free will, human thoughts, or anything related to that. If you want to say the human brain follows the same physics rules as everything else in the universe, and that this implies some indeterminism, sure. That's almost certainly correct. But where do thoughts arise from that?

If you sprinkle randomness on the process, I'll agree with you; but nothing in physics even suggests ANY link between indeterminism, superposition, etc. and thought. So my point still stands: free will as we usually envision it has no reason to exist. The fact that particles can be in two states at once does NOT contradict this.

Indeterminism does not automatically imply free will. None of the open questions of physics, including how qualia arise, could plausibly explain free will. Neuroscience is continuing to show just how deterministic and predictable brains are.

So at this point the burden is on free will proponents to offer a plausible explanation for it. Without resorting to dualism, which is great for religious people but not scientifically useful.

Also, it's funny for you to start this thread with such a disparaging and dismissive comment, then come back and act victimized that you're being questioned. If you are a self aware entity in control of your behavior, maybe you should try harder not to be a hypocritical asshole.

I believe this view of humans is because we work in increasingly deterministic environments (the internet) using predictable methods of communication, text.

We’ve kind of made the world a bit boring and deterministic, places where almost perfect is knowledge is obtainable and so everything feels more and more predictable.

Your day probably consists of using Google, talking on platforms that don’t change much, solving already solved coding problems and communicating with others about office politics problems that we’ve all spoke about over and over again. We literally are just chatbots in this world.

Can you think without words? Can you imagine how it feels to swim, something you might have done, and extend it to fly like a bird? Can you read a novel and feel emotions about characters that don't exist beyond text descriptions of their actions? Language is a way to encode thought. It's not what thoughts are made of.

> Language is a way to encode thought. It's not what thoughts are made of.

What about people who think verbally and don't have a "mind's eye"? In that case, thoughts and their encoding might be closer to 1:1.

Are images and words the only way to think? Are these boundaries even fundamental or are they arbitrary? How does a blind deaf person think? Do you ever think in terms of sensations or movements? Can you remember a roller coaster ride, a car trip? Can you think of a taste?

I find hard to conceive of people who can think only in terms of language.

Does that reasoning work when applied to what you just wrote? If you’re just predicting the next word, then what makes you confident in the conclusion that you’re just predicting your next word?

Yes. This was my second draft. I reviewed my first draft which became an input in my second draft which gained more clarify after reviewing the first draft.

The way people usually say it is that you revised your first draft upon reflection, because it didn't quite get across the point you wanted to make, or something to that effect. So what makes the way you state it more true than the way people typically state it?

I can switch from word generation and evaluation. I saw the score of 1st draft was below my target. And I used the insights from the eval of the 1st draft to inform the 2nd draft.

But you and your brain are much more than the language-forming aspect.

Yes, most of the brain is a glorified microcontroller for running and interpreting data from a billion skin sensors and driving hundreds of muscle actuators, doing depth and object inference from binocular images, locating sounds from the twin microphone array, etc. Language and especially reasoning is a very small part of it all.

Language is the compression of our entire human experience into words, so I'd argue language captures the majority of the thinking aspect of the human experience.

Reply on news.ycombinator.com