Hacker News Reader

Comment by LASR | Hacker News Reader

Often the "stochastic parrot" line is used as a reduction on what an LLM truly is.

I firmly believe that LLMs are stochastic parrots and also that humans are too. To the point where I actually think even consciousness itself is a next-token predictor.

Where the industry is headed - multi-modal models. This really I think is the remaining frontier of LLM <> Human parity.

I also have a 15 month old son. It's totally obvious to me that he's definitely learning by repetition. But the sources of training data is much more high bandwidth than whatever we're training our LLMs on.

It's been a couple of years since GPT-3. It's time to abandon this notion of "stochastic parrot" as a derogatory. Anyone stuck in this mindset really is going to be hindered from making significant progress in developing utility from AI.

View on HN

> I firmly believe that LLMs are stochastic parrots and also that humans are too. To the point where I actually think even consciousness itself is a next-token predictor.

Almost every time I'm on hackernews I end up baffled by software engineers feeling entitled to have an unfunded opinion on scientific disciplines outside of their own field of expertise. I've literally never encountered that level of hubris from anyone else. It's always the software people!

Consciousness is far from being fully understood but having a body and sensorimotor interactions with the environment are already established as fundamental preconditions for cognition and in turn consciousness.

Margaret Wilsons paper from 2002 is a good read: https://link.springer.com/content/pdf/10.3758/BF03196322.pdf

peace

I suspect you don't know what OPs field of expertise is. I also doubt OP would disagree with that statement that the only conscious things we know of have a body and sensorimotor interactions with the environment.

The befuddlement goes even farther for me. LLMs are, effectively, black-box systems that interface with the world via a stochastic parrot interface.

You'd think that software engineers would be a group that easily understands how making radical assumptions about implementation details when looking at nothing but an interface is generally misguided.

I'm not saying that there isn't a strong case to be made against LLMs being intelligent. It's pointing at the stochastic parrot as evidence enough in of itself that confuses me.

Ah, but HN is a platform for not just any software engineer, but the entrepreneurial type.

Which is to say, the type that will believe anything if it's currently easy to hype to VCs.

> You'd think that […]

As a stochastic parrot I'm unable to do that.

Are you saying that ie. paralyzed people don't have consciousness?

First of all, paralyzed people do have bodies. And sensorimotor functions.

Second of all, it wouldn't matter if individually they did or did not. The species does and now our species has developed consciousness. It's part of the package.

If you wanted a counterexample, you should look to plant life. There is some discussion on whether or not plant systems have a form on consciousness. But, then again, plants have bodies at the very least.

What about people with amputations or missing body parts due to genetics - are they less conscious?

There is a spectrum between conscious and unconscious. You could say that under general anesthesia you are a 0/10 on the conscious scale, asleep is 1 or 2, just woken from sleep is maybe 3.... and up to a well rested, well fed, healthy sober adult human near the top of the scale. These are common sense notions of objective consciousness and they are testable and noncontroversial outside of a philosophy argument.

Does this make sense as a rebuttal to your reductio argument?

People dream under general anesthesia, which is a form of consciousness.

I don't think ie. Stephen Hawking was on "lower spectrum of consciousness" either.

To be fair, it's any of the exalted professions that the blessed extend their expertise to. Doctors, lawyers, software engineers. they (we) start with the notion that I'm a smart person, so from first principles, I can conquer the world. nevermind that that's an existing body of work, with their own practitioners to build off of.

Yeah, it's only software people. No one else has unfounded opinions.

But... a parrot has a body. And sure, you'll say "they don't literally mean parrot".. but it's a vague term when you unpack it, and people saying "we are stochastic parrots" are also making a pretty vague comment (they clearly don't mean literally). Anyone who has a small child and understands LLMs is shocked by how much similar they seem to be when it comes to producing output.

The question is: does polly really want a cracker?

Some of us who believe that humans are at least partly statistical parrots have PhDs in relevant fields- for example, my PhD is in Biophysics, I've studied cognitive neuroscience and ML for decades, and while I think embodiment may very well be a necessary condition to reproduce the subjective experience of consciousness, I don't think "having a body and sensorimotor interactions are established as fundamental preconditions for cognition and in turn consciousness". Frankly I think that's an impractical question to answer.

Instead, I work with the following idea: it seems not unlikely that we will, in the next decade or so, create non-embodied machine learning models which simply can't be told apart from a human (through a video chat-like interface). If you can do that, who really cares about whether it's conscious or not?

I don't really think philosophy of the mind is that important here; instead, we should treat this as an engineering problem where we assume brains are subjectively conscious, but that's not a metric we are aiming for.

> software engineers feeling entitled to have an unfunded opinion on scientific disciplines outside of their own field of expertise

There's an XKCD about this behavior[0]. The title is actually "Physicists", but I also have seen it on HN (especially with psychology).

[0]https://xkcd.com/793/

Well with psychology it’s more fair. Thanks to the replication crisis we can say with a straight face that psychologists aren’t even experts on psychology. As usual the demonstrable expertise in the field lies with the pragmatic types. For psychology that means salesmen, pickup artists, advertisers, conmen, propagandists, high school queen bees, and so on.

This is known as the "Why don't you just assume a spherical cow?"

Embodiment is an intellectual dead end in explaining consciousness/sentience. Sure, its relevant to understanding human cognition as we are embodied entities, but it's not much relevant to consciousness as such. The fact that some pattern of signals on my perceptual apparatus is caused by an apple in the real world does not mean that I have knowledge or understanding of an apple in virtue of this causal relation. That my sensory signals are caused by apples is an accident of this world, one we are completely blind to. If all apples in the world were swapped with fapples (fake apples), where all sensory experiences that have up to now been caused by apples are now caused by fapples, we would be none the wiser. The wide content of our perceptual experiences is irrelevant to literally everything we know and how we interact with the world. Our knowledge of the world is limited to our sensory experiences and our deductions, inferences, etc derived from our experiences. Our situatedness in the world is only relevant insofar as it entails the space of possible sensory experiences.

Our sensory experience is the medium by which we learn about the external world. We learn of apples not because of the redness of the sensory experience, but because the pattern of red/not-red experience entails the shape of apples. Conscious experience provides the medium, modulations of which provide the information about features of the external world. It is analogous to how modulations of electromagnetic waves provides information about some distant information source. Understanding consciousness is an orthogonal matter to one's situatedness in the world, just like understanding electromagnetic waves is orthogonal to understanding the information source being modulated into them.

> having a body and sensorimotor interactions with the environment are already established as fundamental preconditions for cognition and in turn consciousness.

Does someone who is blind is not conscious then? How about when someone who is paralysed? or deaf? or someone with low IQ or mental illness?

Stephen Hawking was paralysed most of his life but was a smart guy. If he was not only paralysed but also blind and deaf he would be smart guy still.

I don't think AGI needs to have a body, sensorimotor interaction or even vision to be conscious. We need those for training - if you would be blind, paralysed, deaf from the beginning it would be hard for you learn anything and interact in any way.

Machines have 6th sense that humans don't have - kind of a telepathy where they can exchange tokens/thoughts with different machines or humans much faster than we humans can type or speak.

I've been thinking about this for a while now but I've been approaching it from the opposite direction.

If we were attempting to put someone into some sort of Matrix like reality simulator but we lacked the technology to provide a perfect simulation what level of simulation would be 'good enough' that a human would consider it reality and be able to develop into something we could relate to?

If you gave someone the Helen Keller level of experience, but with reduced tactile sensation, how much could you reduce that touch sensation before they wouldn't be like us?

> If we were attempting to put someone into some sort of Matrix like reality simulator but we lacked the technology to provide a perfect simulation what level of simulation would be 'good enough' that a human would consider it reality and be able to develop into something we could relate to?

Have you tried VR before? You really don't need perfect simulation to be fooled. Good enough is already here, albeit for a short amount of time.

From the first line of Wilson's paper:

> There is a movement afoot in cognitive science to grant the body a central role in shaping the mind.

It is far from true to say "having a body and sensorimotor interactions with the environment are already established as fundamental preconditions for cognition and in turn consciousness"

It is a popular idea in some groups of people that study these questions. But there are many other similar groups of peopel studying these questions who do not agree with it, certainly not stated as strongly as you have put it here.

Also, a reminder that HN readership and its commentariat, while dominated by SWE, is not limited to them.

I'm in the same boat. It feels wrong to contemplate that our consciousness might not be a magical independent agent with supernatural powers, but is rather an emergent property of complex-but-deterministic actions and reactions.

Like it somehow diminishes us. Reduces us to cogs and levers and such.

But I can't imagine how it could be otherwise, though. I'm still baffled by the existence of qualia, phenomenology, etc. Awareness. But bafflement on that front isn't a good reason to reject the possibility that the only thing that separates me from a computer is the level of complexity. Or the structure of the computation. Sometimes things are just weird.

> emergent property of complex-but-deterministic actions and reactions

I think you mean non-deterministic. The last century of physics was dominated by work showing how deterministic systems emerged from non-deterministic foundations. It seems that probability and statistics were the branches of maths behind everything. Who would have thought.

Thanks. I did actually mean to use "deterministic", but only as it sits in opposition to "free will". Is there a better word for what I meant?

Of course there is randomness as well. So, yeah, I should clarify: We don't impose upon the world any kind of "uncaused cause", even if it feels like we do. Everything we think and do is a direct result of some other action. Sometimes that action can trace its lineage to a random particle decay. (Maybe—ultimately—all of them can?) Maybe we even have a source of True Randomness inherent in our minds. But even so, that doesn't lend any support to the common notion of our minds and consciousnesses as being somehow separate from the physical world or the chains of information that run through everything.

I get what you are saying. I was just thinking about the stochastic parrot analogy for consciousness, but I see your comment is more about there not being special sauce to conciousness. But hey, the fact that such behaviour can emerge from simple processes is still pretty damn cool.

I think "deterministic" is the more correct option.

Yes, of course deep down there's unimaginable numbers of atoms randomly bumping around. But so far everything suggests that biological organisms do their damnedest to abstract that away and operate on a deterministic basis.

Think like how you keep changing -- gaining and losing cells, changing chemical balance, and on the whole it takes a lot of damage, even brain damage, to produce measurable results.

I don't think we're ever going to arrive at a satisfactory answer for where qualia comes from, for much the same reason it's impossible to test the quantum suicide hypothesis without actually putting yourself through it enough times to convince yourself statistically.

The only "real" evidence of qualia you have is your own running stream of it; you can try to carefully pour yourself, like a jug of water, into the body of another, and if you go carefully enough you may even succeed to carry that qualia-stream with you. But it's also possible that you pour too quickly, or that you get bumped in the middle of the pouring, and poof - the qualia is just gone. You're left with a p-zombie.

Or maybe not. Maybe it comes right back as soon as the transfer is done, like the final piece of a torrented movie. The important part is you won't know unless you try - and past success never guarantees future success. Maybe you just got lucky on the first pour.

Indeed our consciousness is an emergent property from a complex system, however the complexity of that system - the human brain - is almost unfathomably beyond the complexity of anything we might make in silicon now or in the foreseeable future.

For example, it is possible to write down a (natural/whole) number that completely describes the state of an LLM - its connections and weights etc - for example by simply taking the memory space and converting into a number as a very long sequence of 1s and 0s. The number will be very big, but still indescribably smaller than a number that could perfectly describe the state of a human brain - not least as that number is likely to lie on the real line, or beyond, even if it was calculable. See Cantor for a discussion on the various infinities and their respective cardinalities.

Under the assumption that human brains are limited by our current understanding of physics, their state is finite - infinities are forbidden by definition.

( See the relationship between information theory and Heisenberg's uncertainty principle that results in the law of paraconservation of information : http://www.av8n.com/physics/thermo/entropy-more.html#sec-pha... )

Not true. The idea that QM discretises our universe is incorrect.

> I firmly believe that LLMs are stochastic parrots and also that humans are too. To the point where I actually think even consciousness itself is a next-token predictor.

I agree with the first sentence but not with the second one. Consciousness most probably does not arise from just a next-token predictor. At least not from an architecture similar to current LLMs.

Both humans and LLMs basically learn to predict what happens next. However, LLMs only predict when we ask them. In contrast, humans predict something all the time. Even when we don't have any sensory input, our brain plays scenarios. Maybe consciousness arises because the result of our thinking is fed back as input. In that sense, we simulate a world that includes us acting and communicating in that world.

Also noteworthy, the human brain handles a variety of sensory information and it's output is not only language. LLMs are restricted to only language. But to me it seems like it's enough for consciousness if we can give it the self-referential property.

In order to predict what happens next we need to create a model of the world. We exist as part of the world so we need to model ourselves within it. We also have to model our mind for it to be complete, including the model of the world it contains. Oops, I just created an infinite loop.

Not necessarily infinite. It stops when there's reasonable accuracy. Similar to how we would implement this in software.

> However, LLMs only predict when we ask them. In contrast, humans predict something all the time. Even when we don't have any sensory input, our brain plays scenarios.

Pretty easy to do this exercise with an LLM. At least, easier than building an LLM in the first place. Leave it running, let it talk to itself, revisit partial memories, and explore noise. Really a duct-tape problem here more than anything.

LLMs don't create new information, they only compress existing complexity in their train and inference data sets.

Humans definitely create new information. (Well, at least some humans do.)

Do they though? Or do humans just combine things they have learned about the world?

How are you defining "create new information" ?

In the information theoretic sense. (Increasing information entropy.)

What do you define as the system here? The LLM itself? The world of all text? What about for the human?

Lossy compression + interpolated decompression = new information

Lossy compression decreases information entropy by definition.

Interpolated decompression increases information entropy by definition.

Do we?

Humans can observe new information, but that's obviously not that unique. We can reason about existing information, creating new hypotheses, but that is arguably a compression of existing information. When we act on them and observe the effects they become information, but that's not us creating the information (and LLMs can both act on the environment and have the effects fed back to their input to observe, so it's not really unique).

There is this whole field of art, but art is constantly going through a mental crisis whether anyone is creating anything new, or if it's all just derivations of what has come before. Same with dreams, which appear "novel" but might just be an artefact of our brain's process that compresses the experiences of the day.

Life creates novelty. Humans are a member of that category, but all life is producing novelty.

More information: https://www.nature.com/articles/s41586-023-06600-9

Is the inverse true? Does novelty create life? Can something be novel devoid of life?

let's say you wanted to count the number of coins on a table

you organize them into piles of ten

this created new information

More like you converted information you had about energy powering your muscles into that one - resulting in less total information for you in the end.

that's not how information theory works

I'm not arguing semantics; there's a large amount of math and theory here.

LLM's are basically lossy compressors, they decrease information entropy.

Humans increase information entropy whenever we do anything.

(Some) information entropy is potentially very valuable, so in effect LLMs destroy value by design, while humans can potentially create value. From an economic point of view humans can never be replaced by an LLM.

I agree. A thought experiment I had recently:

Let's say we could somehow train an LLM on all written and spoken language from the western Roman civilization (Republic + Western Empire, up until 476 AD/CE, just so I don't muddy the experiment with near-modern timelines). Would it, without novel information from humans, ever be able to spit out a correct predecessor of modern science like atomic theory? What about steam power, would that be feasible since Romans were toying with it? How far back do we have to go on the tech tree for such an LLM be able to "discover" something novel or generate useful new information?

My thought is that the LLM would forever be "stuck" in the knowledge of the era it was trained in. Something in the complexity of human brains working together is what drives new information. We can continue training new LLMs with new information, and LLMs might be able to find new patterns in data that humans can't see and can augment our work, but the LLM's capability for novelty is stuck on a complexity treadmill, rooted in its training data.

I don't view this ability of humans as some magic consciousness, just a system so complex to us right now that we can't fully understand or re-create it. If we're stochastic parrots, we seem to be ones that are magnitudes more powerful and unpredictable than current LLMs, and maybe even constructed in a way that our current technology path can't hope to replicate.

For your thought experiment, I'd assert that the key missing part is experimentation in the real world, as that is what acquires new information, not the complexity of human brains working together.

If you took millions of genius-level immortal humans with all the same Roman data but had them sit in a blank, empty room with their hands tied and simply discuss philosophy for eternity, I'm certain that they would not be able ever spit out a correct predecessor of modern science like atomic theory. Perhaps they could spit out billions of theories including the atomic theory as well, but they would have no data to presume that the atomic theory is more relevant than any other. Extensive information processing can squeeze out every last ounce of knowledge from some data, but anything that isn't in that data can't be acquired by mere thinking about it. On the other hand, if you gave some "LLM++" the ability to toy around with reality and attempt all kinds of experiments to test various hypotheses, then I wouldn't assume that it would not be forever stuck in the knowledge of the era it was trained in.

Yeah, I like that improvement/clarification. Good assertion. Now I wonder if it changes my stance: are the path modern LLMs are on ever going to replicate this environment for acquiring new information that humans currently operate in?

As usual in these "Yeah, but can AI do this?" threads, the answer is yes, it is already happening: https://www.space.com/mars-oxygen-ai-robot-chemist-splitting...

> Let's say we could somehow train an LLM on all written and spoken language from the western Roman civilization (Republic + Western Empire, up until 476 AD/CE, just so I don't muddy the experiment with near-modern timelines). Would it, without novel information from humans, ever be able to spit out a correct predecessor of modern science like atomic theory?

Funny example - depending on how close the predecessor has to be, the answer is maybe: https://en.wikipedia.org/wiki/Atomism

I disagree they're stochastic parrots, I find othello-gpt very convincing that these models can create world models and respond accordingly.

> Anyone stuck in this mindset really is going to be hindered from making significant progress in developing utility from AI.

I think this specific line shouts out that this is a typical tribalism comment. Once people identify themselves as a part of something, they start to translate the value of that something as their own worth. It's a cheap trick that even young kids play, but can LLM do this? No.

Some might say multi-modal this, train on that-thing, but it already takes tens of thousands of the most advanced hardware and gigawatts of energy to push around numbers to reach where it is. TBH, I don't see it going anywhere, considering ROI on research will decrease as we dig deeper into the same paradigm.

What I want to say is that today's LLM is certainly not the last stop of AI technology, but a lot of advocates tend to consider it as the final form of intelligence. It's certainly a case of extrapolation, and I don't think LLM can do that.

That's a pretty bold statement, coming from someone with the subjective experience of consciousness.

Did you teach your child to crawl? To laugh? To get excited?

Those who don’t understand a concept are doomed to reduce it to concepts they do understand.

I’m currently reading I Am A Strange Loop, a pretty extensive dive into the nature of consciousness. I’m reserving final judgment on how much I agree with the author, but I find it laughable to claim consciousness itself is on the same level as an LLM.

I would have agreed until these recent podcasts that Chomsky did.

Everyone is basically talking out their ass when it comes to language and linguistics. That becomes incredibly obvious listening to Chomsky on chatGPT.

I was even so stupid to think Chomsky wasn't a fan of chatGPT because it somehow invalidated some of his language theories. Low and behold, no, Chomsky actually knows what he is talking about when it comes to linguistics.

What recent podcasts? Can you link to some?

Reply on news.ycombinator.com