Hacker News Reader

Comment by marviel | Hacker News Reader

Author here -- I think it is a stochastic parrot, don't get me wrong.

But my argument is that humans are also, at some level, sufficiently advanced stochastic parrots. At least, as it pertains to many creative endeavors.

You're always building on the back of something that comes before.

We're all just remixing ideas we've heard before.

I mean even this comment -- none of the words I'm saying are new, and many of the word combinations I'm using have been used time and time again. The ideas I'm expressing have merit -- but are they wholly original?

Not really. We all need eachother's creative energies to do our best work.

View on HN

The difference between humans and sophisticated stochastic parrots is reason. The most common kinds of mistakes that chatGPT currently makes are when it says things that are not simply wrong, but don't make sense. Perhaps it will be possible to emulate reason with enough data, training, parameters, etc, but without some representation of the ability to understand what you don't know, what you know, and what follows from those things consistently, I wonder if these kinds of models will ever become truly reliable.

Although even in this example it does say some things that are simply wrong:

"Some programming languages like TypeScript and Swift use a colon for both variable type annotations and function return types."

This is incorrect for Swift, which uses the same arrows as "TenetLang" for function return types. Actually the first thing I thought looking at the example code was "looks swifty but not quite as well designed."

> some representation of the ability to understand what you don't know, what you know, and what follows from those things consistently

Right, that is truly what the chatbots are lacking. They can fool some of the people some of the time, but they can't fool all of the people all of the time.

Their creators - not the chatbots themselves - are trying to "fool" people into believing that the chatbots would have as you say "ability to understand what you don't know, what you know, and what follows from those things consistently".

It's totally possible that humans don't do reason. It's possible that the parrot in our brain makes the decision, and then the "frontend" of our mind makes fake reasons that sound logical enough.

But it's just a possibility, and I don't find it's particularly convincing.

Look up split brain experiments [1]. Basically, in patients where the corpus callosum is severed to some degree, the two halves of the brain have limited communication. Since the two halves control different parts of the sensory system, you can provide information selectively to one or the other half of the brain, and ask that part of the brain to make choices. If you then provide the other half of the brain the wrong information, and ask it to reason about why it made a choice, the other brain will happily pull a ChatGPT and "hallucinate" a reason for a choice neither that half nor the other half of the brain ever made.

While that does not prove that we never apply reasoning ahead of time, it is a pretty compelling indication that we can not trust that reasoning we give isn't a post-rationalisation rather than an indication of our actual reasoning, if any.

[1] https://en.wikipedia.org/wiki/Split-brain

No but that does happen a lot of the time. The difference is that we can choose to engage the deductive engine to verify what the parrot says. Sometimes it's easier to do so (what's 17×9?) and sometimes it's harder (a ball and a bat costs $1.1 and the bat costs $1 more than the ball, what does the ball cost?)

You can ask ChatGPT to show it's working too. It's likely, for many things, (as it is with humans a lot of the time) that the way it does it when it walks you through the process is entirely different to how it does it when it just spits out an answer, but it does work. E.g. I asked it a lot of questions about the effects of artificial gravity using a rotating habitat a few days ago, and had it elaborate a lot of the answers, and it was able to set out the calculations step by step. It wasn't perfect - it made the occasional mistakes - but it also corrected them when I asked follow-up questions about inconsistencies etc. the same way a human would.

(Funnily enough, while both your example questions are easy enough, I feel I was marginally slower on your "easy" question than your "harder" one)

Maybe we're getting close to what makes me doubt the utility of LLMs. Much like humans, they are quick to employ System 1 instead of System 2. Unlike humans, their System 1 is so well trained it has a response for almost everything, so it doesn't have a useful heuristic for engaging System 2.

For me the utility is here today. Even if I have to carefully probe and check the responses there are still plenty of things where it's worth it to me to use it right now.

Fast thinking/slow thinking by Daniel Kannerman (and the Unlocking Project) is a book people need to read and understand. When we slow think, we work things out, when we fast think we are parrots.

Hmm. If you want to argue that ChatGPT has half of what an AI needs, I could buy that a lot more than I buy "ChatGPT is the road to AGI".

Do inference engines have the other half? Or Cyc plus an inference engine? Can that be coupled to ChatGPT?

My own (completely uninformed) take is that such a coupled AI would be very formidable (far more than ChatGPT), but that it will be very hard to do so, because the representations are totally different. Like, really totally - there is no common ground at all.

I didn’t say half. I just said we have 2 ways of thinking.

I just meant "half" in the sense that there are two major chunks that are needed. I did not mean that each "half" was used as much as the other, or was as much work to implement.

I’m sure a stochastic parrot model could be trained to exhibit reasoning, but the issue is that there isn’t any automated loss function which can discern whether the output of a large language model exhibits reasoning or is illogical. When you train based on text similarity, it will have a hard time learning logic, especially given the amount of illogical writing that is out there.

Hmm. That's classically the job of an educator - to choose good training material, not just to let students read the internet all day. Would a ChatGPT trained on a carefully curated reading list do better than one trained on a wider reading list?

But it's more than that. A good educator teaches students to evaluate sources, not to just believe everything they read. As far as I can tell, ChatGPT totally lacks that, and it hurts.

And our reason is controlled by emotions, ego, intellect, etc, etc: our sentience.

Llm’s are the language model. That is it.

Now we have gone down this path to a very useful tool what’s next is connecting it to something that can understand context and memory.

It’s not so hard to imagine being connected to a knowledge graph of all the things and this evolving into a very capable AI.

It’s like Google v1 compared to Google now. Key words versus semantic seo

>The most common kinds of mistakes that chatGPT currently makes are when it says things that are not simply wrong, but don't make sense.

So, like the average voter?

You're being a bit glib, but I tend to agree that a lot of people seriously overestimate the reasoning done by the average human.

I think GPT is on the other hand both over- and underestimated because it speaks well but often makes reasoning mistakes we only expect of someone less eloquent, and it throws us.

If it had come across as an inquisitive child, we'd have been a lot more overbearing of it's "hallucinations" for example, because kids do variations over that all the time.

At the same time, it can do some things most children would have no hope of.

It's a category of "intelligence" we're unfamiliar with, additionally hobbled with no dynamic long term memory.

We have far too insufficient knowledge of how human reasoning work to claim to know we are more than stochastic parrots with vastly more context/memory. It's way too early to claim there's some qualitative difference.

(And humans are not very reliable; more than current models, sure, but still pretty bad)

Except the fact that it does both. It commits mistakes AND it also comes up with remarkable, novel output that makes sense. Perhaps you just haven't seen an example of it that's convincing enough.

https://www.engraved.blog/building-a-virtual-machine-inside/

Here's another one.

I would go even further. People are not even that advanced stochastic parrots. In the similar task, we failed just as spectacularly as the GPT-4 did right now. Rust, Zig, Julia, Kotlin, and dozens smaller "new" languages fail to bear a smear of originality in them. Their authors remixed old ideas in different proportions and got different grammars as a result. That's something an LLM can do and much faster too.

But there is also Unison. That one new language that stands out, and the very fact that it does makes it improbable for an LLM to generate. LM is a language model, it avoids marginality and originality altogether by its design.

Language models can and are useful if you specifically want to avoid marginality. To reduce noise, to remove errors. There is huge potential in them. If the problem was to design the most average programming language with no purpose, no market niche, and no technological context - then GPT-4 is clearly a winner.

> Rust, Zig, Julia, Kotlin, and dozens smaller "new" languages fail to bear a smear of originality in them.

To be fair, most programming languages don't try to be original. They try to solve problems. They also try to keep syntax close enough to other established programming treds so people have an easier time learning them.

> I would go even further. People are not even that advanced stochastic parrots. In the similar task, we failed just as spectacularly as the GPT-4 did right now. Rust, Zig, Julia, Kotlin, and dozens smaller "new" languages fail to bear a smear of originality in them. Their authors remixed old ideas in different proportions and got different grammars as a result. That's something an LLM can do and much faster too.

We are in good part stochastic parrots for sure, but there's a small nuance that is completely lost in your reasoning and which is what makes the difference today between humans and GPT.

In those languages, reusing commonly known syntax is an explicit feature. Whenever the language had no reason to introduce something new, it used known idioms to lower the amount of things one has to learn to use the language.

Occasionally though, a language will have something unusual, but it won't be for random reasons, but instead to mark something unique and interesting that was added to the language to tackle a facet of programming in a new way.

Rust has lifetime annotations, Zig inverts the usage of async/await (https://kristoff.it/blog/zig-colorblind-async-await/) to support color-blindness.

Present day GPT wouldn't be able to come up with a borrow checker or colorblind asyncness all by itself.

> Language models can and are useful if you specifically want to avoid marginality. To reduce noise, to remove errors.

But this makes me think of Shannon Information Theory. What you're describing is something that has no actual information (in the Shannon sense) at all.

And maybe that's why so much GPT output reads so blandly. Even the ones that are not glaringly wrong still read like... like food with no seasoning.

Really? As far as I know, Rust’s borrow checker was a newly developed approach to memory safety. There have been other non-GC/RC approaches to memory safety in e.g. Cyclone or Ada but Rust’s system is novel.

Likewise, Julia’s use of just in time compilation of dynamic dispatched code to achieve zero overhead execution was rather groundbreaking.

Small LMs do tend to avoid originality, but large LMs can use the context as a starting point to walk outside their training data distribution. They can do that by being better at composing concepts and using context-conditioning, which is also known as in-context learning.

It wasn’t an originality contest, so there’s at least one non-sequitur in that statement. Everyone can create an original syntax, given modern character sets and formatting abilities. The key issue with that is the amount of developers who want to learn and switch between these is low.

Thanks for pointing out Unison -- I'm definitely going to check it out.

There is another language in this space I've had my eye on as well, but I can't recall the name at the moment.

> But my argument is that humans are also, at some level, sufficiently advanced stochastic parrots.

But humans have a feedback loop called "consciousness" that sits above the stochastic parrot level, can reason about it, and can separate the wheat from the chaff.

AI still has to reach this level. ChatGPT cannot judge if its answers are right or wrong. It just claims things confidently, and as long as it is parroting correctly, it is right, and that's why humans get fooled into thinking that maybe they can trust it. (That's also why humans get fooled by human bullshitters who claim things confidently).

So if you don't use ChatGPT as what it is, a lookup-tool trained on the content of the internet, which can't even tell you where it got the information from, and the onus is on you to verify if the information is correct or not, and if instead try to use it as some kind of expert you can trust (which I have seen quite a few people do), then you are doing it wrong.

I don't disagree we're still better in a number of ways, but if GPT-4 was given a database, and a "while" loop -- do you think it would gain consciousness?

I think the difference is that LLMs are essentially just stochastic parrots (SP) Humans are stochastic parrots and more. In some ways the SP behavior of LLMs already far outstrips the SP abilities of humans. And there are some leaps that humans' SP ablities will be hard to mimic. But at the end of the day, a human is still more than just an LLM. What exactly, we'll be debating that til the end of time likely. We know that an LLM can probably give us a pretty condensed summary of mankinds philosophical wranglings on the subject. :D

> The ideas I'm expressing have merit -- but are they wholly original?

I think the difference is that your intention is to write sentences which you think and believe are correct and which can help other people understand the subject. Chatbot has no intentions of its own, it only imitates texts it has read from the internet. As far as it is concerned they might be totally wrong. You on the other hand have the ability -- and the desire -- to reason about what you are saying and think whether it is actually true or not.

I'm not sure I can support the idea that the agent doesn't have the ability to reason -- try to give it any complex (text-based) puzzle you can think of and it'll do just about as well as an average person, oftentimes much better.

And I think you may be wrong to say the chatbot doesn't have intentions -- it's intentions based on its training are to accurately predict the next character. It doesn't care in the same way we do, sure -- but you could make a case (and it will be made in courts in the next few years, I don't doubt) that these agents do have desires that are analogous to our own, by the nature of their training process.

I don't know where that leaves us to be honest, but it's an interesting topic to discuss.

You say > it's intentions based on its training are to accurately predict the next character.

then you say: > these agents do have desires that are analogous to our own

I think its very in-human to have a single desire which is to predict the next character. That is not analogous to our desires.

And the intention to predict the next character is not the intention of the chatbot, it is the intention of whoever created the chatbot or whoever is using it for that purpose.

AI is a tool created by humans to fulfill the intentions of those humans.

Is it the intention of a gun to kill people? No, it is the intention of whichever human who uses a gun for such a purpose. Is it the intention of AI to predict the next character? No that is the intention of the human who uses AI for such a purpose.

Clearly, having a single life desire and nothing else is in-human, I completely agree!

The full quote was > but you could make a case (and it will be made in courts in the next few years, I don't doubt) that these agents do have desires analogous to our own

Analogous doesn't mean "the same" it means "somehow similar."

However I would challenge you to consider more specifically why this type of desire is different from our own desires. Besides the biological machinery, what makes this type of desire different from ours?

There is no "desire" in the computer. Therefore the question of whether its desire is different from our desires is meaningless, because it does not have a desire.

Computer just executes instructions. It doesn't matter to it whether its desires are fulfilled or not. Whereas humans do have desires: If we get thirsty we suffer unless we get our desire of drinking fulfilled.

We don't know enough about how the brain work to be able to say that our "intent" is any more than a combination of memory and an after the fact rationalisation of a stochastic process.

We also don't know enough to support the hypothesis that you put forward either (and which I reject, from a philosophical standpoint).

> We're all just remixing ideas we've heard before.

We're largely remixing ideas we've heard before, but not just I think. If you're just remixing ideas that came before... where did those ideas come from?

I think you can get quite far remixing ideas but for more novel concepts to emerge you have to actually create new stuff.

In biology it seems that evolution has found it worthwhile not to minimise mutations too much. In genetic programming you need both mutation and crossover for evolution.

I don't think LLMs really have that mutation of ideas (at the moment!), and I suspect humans create genuinely new concepts in a much more sophisticated way than mere random mutation.

Nope, we parrot mostly, but not always. When we do not, we create something new, and then we started parroting again. Mostly.

The question is, can ai do something new, understand context of the new thing, understand when and where it will be usable and when it make sense to apply it.

Is something new that isn't a synthesis of past things not just a stochastic process?

Human inventiveness seems to heavily follow a pattern of relatively minor iterations of what came before, extraordinarily rarely something that even seems to defy the past and be truly different and independent.

Democracy, nuclear power, calculus, dna helix, x-ray, written word, relativity comes to mind for starters. There are millions others.

Care to elaborate how any of those involved humans inventing something that wasn't a combination of synthesis with a stochastic process filling in the gaps?

E.g the discovery of x-rays took over a century of applying small iterations to established processes to accumulate data. There was no big, sudden leap of insight there.

How is democracy a stochastic process?

Not what I meant. Talking about the process of the inventions, not the inventions themselves.

When we're creating something "new" -- are you positing that it isn't built on top of other ideas, being combined in new ways?

New can be 1. something without a predecessor, 2. synthesis of other ideas that creates totally new application or theory

Can you provide an example of 1?

I think there are definitely some out there that feel almost "singular" in the way you describe, but each time I try to find a non-predecessor item I end up feeling "nope there are still dependencies"

Relativity for instance. New does not exclude dependencies or partial predecessors. New is something that is totally different from before.

Relativity comes from exploring the problem of the incompatibilities of pre-existing theories and experiments. I don't see anything that suggests it requires anything more than synthesis and a stochastic process to explore the gaps and inconsistencies. In terms of timing, they're the culmination of decades of pre-existing work, with e.g. the Lorentz transformations named the same year Einstein published his paper on special relativity.

Lorentz transformation is not relativity. I think we might have a problem with definition and understanding what we try to accomplish with this discussion. If I understand you correctly, then stochastic processes will create an all knowing environment if you give it enough time. I think that is false. Most of the time it is true, but there are points in time where there is a leap in knowledge that are not reachable by just iteration.

> Lorentz transformation is not relativity.

Not the point. The point being that there was a number of precursors, not some sudden insight from nothing.

> If I understand you correctly, then stochastic processes will create an all knowing environment if you give it enough time. I think that is false. Most of the time it is true, but there are points in time where there is a leap in knowledge that are not reachable by just iteration.

This presumes that there's no element of randomness in that stochastic process, which is not a sound assumption.

yeah, but colloquially when people start using the term "stochastic parrot" they're referring to something far stupider. From a Technical and pedantic standpoint it could be said ALL intelligence is isomorphic to a stochastic parrot so it's not reasonable to assume in common communication that people are referring to the technical definition when they talk.

Your point in the article is basically saying that chatGPT is not as stupid as people think and also suggesting the fact that humans can be stupid in a similar way to chatGPT as well.

> yeah, but colloquially when people start using the term "stochastic parrot" they're referring to something far stupider. From a Technical and pedantic standpoint it could be said ALL intelligence is isomorphic to a stochastic parrot

I mean that's somewhat his point (if I understand him correctly). All intelligence could effectively be described as a "stochastic parrot" as a result, calling something a "stochastic parrot" is a roughly meaningless insult that is really a thinly-veiled way of saying "no it's dumber than me" without actually using those words.

And his point is, it's essentially a defense mechanism in people when challenged by something that is much smarter than expected to overly emphasis it's mistakes and fall back to "no it'd dumber than me" than evaluate it for what it actually is.

I think that's the issue. It's a defense mechanism.

If you replace "stochastic parrot" in most of those comments with "it's dumber than me" you see what the comment essentially is. "It's dumber than me. I'm smarter than it, I don't need to be worried about it".

A person who does understand how new technology works will see it as a dumb contraption.

A person who doesn't understand how it works will see it as magic.

Lol no. That's just what you tell yourself to feel smart.

They're are many who understand that don't think of it as dumb.

"I think GPT-3 is artificial general intelligence, AGI. I think GPT-3 is as intelligent as a human. And I think that it is probably more intelligent than a human in a restricted way… in many ways it is more purely intelligent than humans are. I think humans are approximating what GPT-3 is doing, not vice versa.”

— Connor Leahy, co-founder of EleutherAI, creator of GPT-J (November 2020)

A founder trying to sell a product by telling you the product is magic? Yeah, very trustworthy source there.

That quote reveals either that they are 1) delusional or 2) a fraudster. Your comment leans more towards 2) but I think 1) is also a possibility, which in some ways I find more concerning (delusional cult leaders being potentially more dangerous than common fraudsters).

>I mean that's somewhat his point

His point is humans and GPT are both stochastic parrots. I'm saying ALL intelligence even the one in an ant or a chess AI is a stochastic parrot. Thus it's pointless to use this term to compare intelligence. Therefore people must not actually be referring to the technical definition when they use the word "stochastic parrot."

>If you replace "stochastic parrot" in most of those comments with "it's dumber than me" you see what the comment essentially is. "It's dumber than me. I'm smarter than it, I don't need to be worried about it".

Yes this is Exactly What I was saying in the post you replied to. You and I are in agreement.

I’m dumber than you

> none of the words I'm saying are new, and many of the word combinations I'm using have been used time and time again. The ideas I'm expressing have merit -- but are they wholly original?

Then why are you saying it? But besides that query, your "word combination" bit elicited a 'so what?'

> We're all just remixing ideas we've heard before.

Something wrong with the grammar in that sentence. (Or do you go by we/us?)

Reply on news.ycombinator.com