Hacker News Reader

Comment by i-use-nixos-btw | Hacker News Reader

I’d be interested in hearing from anyone who takes the Chinese Room scenario seriously, or at least can see how it applies to any of this.

I cannot see that it matters if a computer understands something. If it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes.

It only matters if you probe beyond the realm at which you previously decided it matters (e.g roasting and eating it), at which point you are also insisting that it walk, quack and TASTE like a duck. So then you quantify that, change the goalposts, and assess every prospective duck against that.

And if one comes along that matches all of those but doesn’t have wings, then if you deny it to be a duck FOR ALL INTENTS AND PURPOSES it simply means you didn’t specify your requirements.

I’m no philosopher, but if your argument hinges on moving goalposts until purity is reached, and your basic assumption is that the requirements for purity are infinite, then it’s not a very useful argument.

It seems to me to posit that to understand requires that the understandee is human. If that’s the case we just pick another word for it and move on with our lives.

View on HN

In my understanding of the Chinese Room example, the resolution to the argument is that the *human* may not understand Chinese, but the *system as a whole* can be said to understand it.

With this in mind, I think asking whether ChatGPT *in and of itself* is "conscious" or has "agency" is sort of like asking if the speech center of a particular human's brain is "conscious" or has "agency": it's not really a question that makes sense, because the speech center of a brain is just one part of a densely interconnected system that we only interpret as a "mind" when considered in its totality.

Good point, that very much vibes with my thoughts on this matter. Lately, I've been contemplating the analogy between the role LLMs might take within society with that of the brain's language center* in human behavior. There's definitely a way in which we resemble these models. More than some might like to admit. The cleverness, but also the hallucinating, gaslighting and other such behaviors.

And on the other hand, any way you'd slice it, it seems to me LLMs - and software systems in general - necessarily lack intrinsic motivation. By definition, any goal it has can only be the goal of whoever designed that system. Even if its maker decides - "let it pick goals randomly", those randomly picked goals are just intermediate steps toward the enacting of the programmer's original goal. Robert Miles' YouTube videos on alignment shed light on these issues also. For example: https://www.youtube.com/watch?v=hEUO6pjwFOo

Another relevant source on these issues is the book "The Master and his Emissary", which discusses how basically the language center can, in some way - I'm simplifying a lot, fall prey to the illusion that "it" is the entirety of human consciousness.

* or at least some subsystems of that language center, it's important to remember how little we still understand of human cognition

What goals do we have that aren't essentially all boiled down to whatever evolution, genetics, and our environment have sorted of molded into us?

If you subscribe to a purely mechanistic world-view, i.e. computationalism, then yes. But that's a leap of faith I cannot justify taking. It's a matter of faith, because though we cannot exclude the possibility logically, it also doesn't follow necessarily from our experience of life, at least as far as I can see. Yes, so many times throughout the ages, scientists have discovered mechanisms to explain things which we've historically been convinced will always be outside the purview of science.

But that doesn't mean everything will one day be explained. And one thing that remains unexplained is our consciousness. The problem of qualia. Free will. The problem of suffering. We just don't understand those. Maybe they are simply epiphenomena, maybe they are false problems. But when it comes to software systems, we know with certainty that they don't have free will, don't experience qualia, pain or hope or I-ness.

Sure, it's a difference that disappears if one takes that leap of faith into computationalism. Then, to maintain integrity, one would have to show the same deference to these models as one shows to their fellow human. One would have to think hard about not over-working these already enslaved fellow beings. One would have to consider fighting for the rights of these models.

> Then, to maintain integrity, one would have to show the same deference to these models as one shows to their fellow human.

Except they’re not even remotely close to anything like human intelligence. As I wrote in another comment they are very capable systems, to the point where in some ways they show some level of elementary understanding, but in many forms of reasoning they are utterly and completely incapable. Assigning human equivalent cognitive status is patently absurd. And yes I am a physicalist and I see no reason why a computer system could not achieve human equivalent cognitive ability. These just aren’t that. They may be an important step towards it though.

They might well be in a couple of years, once they become deeply integrated with symbolic techniques. It's already happening with plugins, chain-of-thought reasoning, self-reflection etc. Soon the illusion will be very convincing and hard to shake off. Yet to me, nothing essential will have changed, and the idea of treating these systems as our equals will remain just as patently absurd as before. I expect this will the physicalist position a much more fraught decision, because it will impose some hard limits on their interaction and use of these technologies on those who subscribe to it.

I don't think that's the correct take for the room. Say the human speaks english. If you asked them what the conversation was about, and they had the full resources of the room at their disposal could they tell you? No, because the room doesn't actually allow them to understand chinese, it's just a symbol lookup table. The lookup table doesn't mean the system understands chinese, just the relationship between symbols that can lead to a coherent output.

What if the human learns all the rules? Then the system as a whole is the human.

Here's an example that I think garners more agreement that properties of a limit ("really understanding") don't necessarily mean that any path towards that limit has the properties of the limit. I think there's a lot of room for disagreement about whether this is a factually-accurate analogy and I'm not trying to argue either way on that, just trying to answer your question about how one might make these sorts of arguments if one has a certain belief about the facts.

Let's say we're trying to build a calculator that only needs to do integer addition. And we decide to build it by building a giant if-else chain that hardcodes the answer to each and every possible addition. And due to finite resources, we're going to hardcode all the additions of integers up to absolute value N, but we will increase N over time.

Everything you said applies equally to this situation: it quacks like a duck, and when we talk about things it can't do we have to continually move the goalposts each time a new version comes out. It also has the property that there is a "scaling law" that says that each time you double N you get predictably better performance from the system, and you can do this without bound and continually approach a limit where it can answer any question indistinguishably from something we might call "true understanding".

But I think it's a bit easier to agree that in this case that it's not "really doing" addition and is a bit short of our wish to have an artificial addition system. And if someone touts this system as the way to automate addition we might feel a bit irritated.

Again, many people will say that this is a bad analogy because LLMs operate quite differently, and I'm not trying to argue for or against that. Just trying to give my explanation for how a certain understanding of the facts can imply the kind of conclusion that you are trying to understand.

I think what this points towards is that we care about the internal mechanism. If we prod it externally and it gives the wrong answer, then the internal mechanism is definitely wrong. But if we get the right answers and then open it up and find the internals are still wrong, it's still wrong.

This illuminates a contradiction: the walks like a duck thing is incompatible with the internals being a duck. If you see a creature with feathers that waddles and can fly, it might still be a robot when you open it. So your test cannot just rely on external tests. But you also want to create a definition of artificial intelligence that doesn't depend on being made of meat and electricity.

> I think what this points towards is that we care about the internal mechanism.

The mechanism is what makes a system interesting.

In software this is why we develop libraries of algorithms and code we can reuse and compose into new solutions. The programmer is providing the intellectual flexibility, while the code is the set of capabilities. It’s why this is a superior approach, compared to building a single monolithic mass of procedural code from scratch in a single variable scope for every program we write.

Solutions matter because it’s not just about what a system can do now, it’s about what it can learn or be adapted to do next.

I think @dvt's comment above is a good attempt at answering this question. I agree with him that intrinsic motivation and a capacity for suffering, hope and all the other emotions (which we share with pretty much all animals, if not plants too) are at the top of the list. Cleverness is there also, but not at the top of the list.

re: Calculator examples, I kinda see them both as information systems that achieve a result but there's a phase difference in where the information is stored.

Similar to how space is 4D such that with relativity going faster in a spatial dimension kind of "borrows" from the time dimension (in a hand wavy way).

By analogy, you can have something that's purely a lookup table, or on the other hand, completely based on an algorithm, and the full lookup table is kind of "borrowing" from the algorithmic dimension of the information system space and vice-verse the fully algorithmic version is borrowing from the hardcoded dimension of the information system space.

Under the condition that you're adding integers below N, then if you consider BOTH the (hardcoded, algorithmic) as a singular space (as with 4D space time) then they are equivalent.

Need to work on this theory further to make it more understandable, but I think this way about intelligence.

Intelligence sits as a pattern in the information system space that can range anywhere from hardcoded to algorithmic (if we choose to orthogonalize the space this way). But what actually matters is the system's future impact on it's local laws of physics, and for that purpose both implementations are equivalent.

Edit: Conversation with GPT-4 about this https://sharegpt.com/c/Sbs4XgI

The only thing that separates your mechanism for doing addition from what computers actually do is efficiency. Computers can only add numbers up to some fixed size, e.g. 64 bits, and you have to use repetition to add anything larger.

Does that mean computers are not "really doing" addition?

There’s a lot more different than efficiency. We can program computers with algorithms capable of computing any possible addition, the limitation being only the memory of the computer and time, not the algorithm itself. Those algorithms are genuinely doing addition in a way that a pre-computed lookup table is not. It’s the difference between computing an addition in your head and just remembering that 2 + 2 is 4.

What I find really entertaining is the "just predicting the next token" argument.

If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.

> If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks

But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until the recent internet age, in which garbage survives as well as the gold)).

> then maybe there's a bit of hubris in how smart we think we actually are

I feel like you could say this if ChatGPT or whatever obtained its knowledge some other way than direct guidance from humans, but since we hand-fed it the answers, it falls a little flat for me.

I'm open to persuasion.

chatgpt doesnt just feed us back answers we already taught it. It learned relationships and semantics so it can apply that knowledge to do something novel. For instance, I took the basic of a dream and told it to turn it into a short story. The short story wasn't bad. I said make it more exciting, it updated the story such that one of the cars exploded. I guess chatgpt learned excitement from michael bay.

(I'm going to be brusque for the sake of the argument, I very much could be wrong and I don't even know how much I believe of the argument I'm making.)

> chatgpt doesnt just feed us back answers we already taught it

True, there is some structure to the answers we already taught it that it statistically mimics as well.

> It learned relationships and semantics so it can apply that knowledge to do something novel

Can you provide an example of this novelty? I think we underestimate the depth and variety of things that humans have written about and put on the internet, and so while anything you ask ChatGPT to do might be outside of your own experience, it's highly likely that it's already been thought before and uploaded to the internet, and that ChatGPT is just parrotting back something to you that is very similar to what it has already seen.

This effect of ChatGPT having so much more experience/training data than any single human being such that it can convince any single human that it is original is an interesting one.

This is why I think, for example, that image generation will result in (a period of) "artistic inbreeding." Because there is so much that other humans have done that is outside of any individual's experience, we will accept e.g. Midjourney's output as something moving and original, when in reality it's just a slight variation on something that someone else has done before that we haven't seen.

(Again apologies for any rudeness, I respect your opinion and experiences and am enjoying the conversation.)

I took the basics of a dream I had, and asked it to turn it into a short story. the result was pretty good. Is it using stuff already to seed its responses? sure, but thats what we do to. Nothing you do or say wasn't taught to you. But these are not simply parroting responses.

I said this to chatgpt: I had a dream that me and my friend were in a car accident, and we had a choice in deciding how to use 1 hour. we could use it to live out our last final moments or spend that hour on trying to think of a solution to survive the wreck. turn this into a short story.

As they sat trapped in their overturned car, Maria and her best friend Rachel had a dilemma. They only had one hour left to live and had to choose how to spend it. Rachel suggested they spend their final moments together, talking about all their favorite memories. But Maria had a different idea. She proposed they use the hour to think of a solution to survive the wreck. Rachel was hesitant at first, but ultimately agreed. For the next hour, they brainstormed every possible escape plan they could think of. They tried pushing against the doors, breaking the windows, and even attempting to signal for help. Just as they were about to give up hope, Maria remembered a tool in the trunk of the car. She crawled over to retrieve it and used it to pry open the door. Rachel followed her lead, and they finally escaped the car just in time. As they stumbled away from the wreck, both girls were in shock that they had made it out alive. They hugged each other tightly, grateful for the hour they spent trying to find a solution instead of giving up. From that moment on, they made a promise to never take a single moment for granted.

> Nothing you do or say wasn't taught to you.

If nothing we do or say wasn't taught to us then where did all human knowledge come from in the first place? This doesn't hold up.

(Again, being direct for the sake of argument, please forgive any unkindness.)

From our environment, genetics, and other people. We simply are able to take in more inputs (i.e. not just text) than LLMs.

I would agree that much more than we're usually ready to admit to ourselves is second-hand, but saying everything is going too far. Inventions and discoveries are happening all the time, at all scales.

Inventions and discoveries are basically a very advanced form of pattern recognition - seeing "interesting" regularities in some particular input where no-one has spotted them before.

And LLMs are capable of making discoveries in this sense, if you feed them data and ask to spot the regularities. They're not particularly good at it, but that's a different question from whether they're able to do it in principle.

Yes, in as far as LLMs can be said to make inventions and discoveries, this is clearly how they do it. And yes, these type of processes definitely play a big part in our human creative capacity. But to say this is all there is to it, is going too far in my opinion. We just don't know. There's still so much we don't understand about ourselves. We haven't designed ourselves after all, we just happened to "come to" one bright primeval day and since then we've been exploring and discovering ourselves. And again and again, we've found ourselves in that seductive spot where we thought we "got it" and there's nothing essential about ourselves we're still missing, only again and again to be proven wrong. Dogmas crumble, new dogmas coagulate, only to dissolve again. Or, we could use the more dignified word "paradigm".

I mean, to me at least, that is the definition of discovery. The exact process used to spot the pattern is an implementation detail.

And yes, I agree that we really just don't know too many things. But my impression is that we're overestimating just how complicated out behavior really is.

The rabbit hole goes very deep with these questions. For example, you left out above the other half of the equation: inventions. Our creative ability. Is that just more pattern recognition? And can discovery and invention be always cleanly teased apart? Also, what humans might have access to is something that is more simple than we imagine. Mystics and philosophers have tried to point towards it. One book that discusses these things in the context of western science and philosophy is Nature Likes to Hide: https://www.amazon.com/Nature-Loves-Hide-Quantum-Perspective...

I would argue that invention is the same thing, yes - identifying patterns in the environment that can be exploited for productive purposes.

FWIW I think it's not a coincidence that LSD - which kicks pattern matching capabilities of the brain into high gear, so to speak - is commonly used as a drug to boost both "creativity" and "inventiveness".

Yes, but that dream? It could never have it. Sure, it can produce at times very convincing descriptions of supposed dreams, but not actually have the experience of dreaming. Because of that, there will always be ways it will eventually miss-step when trying to mimic human narratives.

Sorry about this, but I couldn't resist:

GTP4, rewrite the above message to be less brusque

I hope you don't mind me sharing a different perspective for the sake of discussion. I might be mistaken, and I'm not entirely sure how much I believe in the argument I'm presenting.

It's true that ChatGPT doesn't only provide answers based on what it has been taught, but it also uses the structure of those answers to create statistically similar responses. However, when it comes to demonstrating novelty, I think we might be underestimating the vast amount of information and variety that humans have written about and shared online. While anything we ask ChatGPT to do might be new to us personally, it's highly likely that it has already been thought of and documented online, and ChatGPT is simply providing a similar response based on its prior knowledge.

This phenomenon, where ChatGPT has significantly more training data and experience than any single human, leading to the illusion of originality, is quite intriguing. For instance, when it comes to image generation, we might experience a period of "artistic inbreeding" because we, as individuals, are not aware of everything others have done. We may perceive something like Midjourney's output as moving and original, when in reality, it could just be a slight variation of someone else's work that we haven't seen before.

Please don't take this as me being confrontational; I genuinely respect your opinion and experiences, and I'm enjoying our conversation.

> But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history

Sure, I was saying "better" in the sense that if for X task, it can do better than Y% of humans.

> since we hand-fed it the answers, it falls a little flat for me

We didn't really hand-fed it any answers though did we? If you put a human in a white box all its life, with access to the entire dataset on a screen but no social interaction, nothing to see aside from the text, nothing to hear, nothing to feel, nothing to taste, etc, it'd be very impressed if they were then able to create answers that seem to display such thoughtful and complex understanding of the world.

I think the human would make a lot of the same fundamental errors LLMs make, for similar reasons. The level to which LLMs seem to understand the world is highly superficial because it is entirely linguistic. Also human written texts about the world and human affairs miss out huge swathes of contextual information that we safely assume actual humans have. LLMs don’t have any of that, which is why they fall flat on their faces in so many ways.

Absolutely. What’s fascinating is that they’re getting such good understanding of many things through just text. Multimodal models that can process text, images, sounds, video, etc. are gonna be very interesting for that very reason

I think it’s undeniable that LLMs encode knowledge, but the way they do so and what their answers imply, compared to what the same answer from a human would imply, are completely different.

For example if a human explains the process for solving a mathematical problem, we know that person knows how to solve that problem. That’s not necessarily true of an LLM. They can give such explanations because they have been trained on many texts explaining those procedures, therefore they can generate texts of that form. However texts containing an actual mathematical problem and the workings for solving it are a completely different class of text for an LLM. The probabilistic token weightings for the maths text explanation don’t help at all. So yes these are fascinating, knowledgeable and even in some ways very intelligent systems. However it a radically different form of intelligence from us, in ways we find difficult to reason about.

Well it's like birds and airplanes. Do airplanes "fly" in the same sense that birds do? Of course not, birds flap their wings and airplanes need to be built, fueled and flown by humans. You could argue that the way birds fly is "more natural" or superior in some ways but I've yet to see a bird fly Mach 3.

If you replace the analogy with humans and LLMs, LLMs won't ever reason or understand things in the same way we do, but if/when their output gets much smarter than us across the board, will it really matter?

I think the issue is there are good reasons to think LLMs architected and trained the way they are now can never approach human reasoning capability. That’s because the corpus of human written material is simply grossly inadequate to communicate or encode the knowledge necessary for that.

Our written material assumes huge swathes of contextual knowledge, real world experience, and human lived experience that LLMs don’t and can’t have. At least architected and trained as they are now.

Thats on top of the crippling inability LLMs have to generalise an ability to perform a task from the ability to generate a description of how to do the task. Plus many other similar limitations that would be inexplicable if displayed by a human.

Of course LLMs aren’t the final word in AI development. I think they’re a vitally important step towards general AI, and we’ll get there eventually as we develop ever more capable architectures.

> LLMs architected and trained the way they are now can never approach human reasoning capability

Not sure if you’ve played with GPT-4 but honestly it’s getting there. If you take the bar exam, ChatGPT was in the bottom 10% of participants, GPT-4 is in the top 90%.

It obviously isn’t the ultimate test of reasoning/intelligence but I think we would agree that a human who’s in the top 90% is likely to be pretty smart.

> Of course LLMs aren’t the final word in AI development

Couldn’t agree more. AGI will come from plugging a few of these systems together.

GPT4 still suffers from the same limitations I outlined earlier though. For example that being able to explain how to do things is independent of being able to actually do them. That’s a crippling cognitive limitation. This is just not as obvious because for some tasks it’s been trained how to do them through different methods.

Let’s imagine a map of cognitive capabilities. Humans are a big area on that map. Previous AI systems were small dots or lines on that map, some of them like AlphaZero extending outside the human zone. ChatGPT is an archipelago of several decent sized blobs disconnected from each other, and some of those edge out lightly outside the human Zone. It’s better at some specific tasks than humans.

The problem is the sometimes large gaps between some of the blobs. Capacity at some tasks tell you nothing about its ability at what we would think of as closely related tasks for a human. For GPT4 even, these are utterly different tasks and if it can do them both, it can often do them for completely different reasons than a human does.

If you test it at say 10 tasks that all happen to fall within its capabilities, those widely separated blobs of ability, you’d think it was incredibly intelligent at a huge range of tasks, unaware of the gaps. With a human you’d know those areas would be connected. But with GPT they are not. It’s by probing the gaps where it fails that we begin to understand how much and in what ways it fundamentally differs from us.

This map is getting harder for outsiders to probe though, because OpenAI is papering over some gaps with tuned training. This is like adding some new blobs in a different colour. These appear to close some gaps and add new capabilities, but the systems in the model that implement those aren’t related to the features of the model that give it its other abilities.

Thing is, they can still solve the problem, even if the problem was not one from its training set.

And, more importantly, they solve the problem much better if you tell them to reason about it in writing first before giving the final answer.

Yes I know, as I said they are very knowledgeable and in some ways very intelligent. We just need to bear in mind their processing architecture is radically different from our. This makes our intuitions about their abilities highly error prone.

Absolutely. The shoggoth metaphor is extremely apt here.

What I was specifically responding to is the claim that they can only solve certain kinds of problems because those kinds of problems (and their solutions) were in the training set. By now there's plenty of counter-examples of unique problems that are nevertheless solved. At which point I think we do have to call it "understanding" and "reasoning", even as we acknowledge that it is a very alien form of understanding and reasoning that we just barely managed to squeeze into something that kinda sorta feels humanish.

simonh says >"We just need to bear in mind their processing architecture is radically different from ours."<

The hardware architectures are certainly different but there is a possibility that at least parts of the "software" architectures may be remarkably similar.

Time and research will tell.

[0]if we get a bit quantum (or God for some), then backtracking could happen by collapsing the dead-ends and "changing" history to stay with what turns out to be the solid plan. Could emergent conscience on AI's neurons do the planning and reasoning that it rather seems to be doing but ML experts will say it is not? If our conscience could by any chance reside not in the electrical currents of the wetware, could AI's reason also not reside in tokens? Is there some mysterious process possibly taking place and will philosophy probe it?

0: pasted from another thread

We've known for a long time that computers can do calculations far, far, far faster than us.

We continue to figure out new ways to make those calculations do more complicated things faster than humans.

What is intelligence beyond calculation is an ancient question, but not the one I'm most interested in at the moment, re: today's tools.

I'm curious right now about if there's meaning to other people in human creation vs automation creation. E.g. is there a meaningful difference between an algorithm curating a feed of human-made TikTok videos and an algorithm both curating and creating a feed of human-made TikTok videos.

Both qualitatively in terms of "would people engage with it to the same level" and quantitatively in terms of "how many new trends would emerge, how would they vary, how does that machine ecosystem of content generation behave compared to a human one" if you remove any human curation/training/feedback/nudging/etc from the flow beyond just "how many views/likes did you get?"

I think as soon as text2video gets really good (like midjourney level), there’s gonna be so much AI generated content that unless it’s all extremely good, human made content will be something people search specifically for.

As for curation, I think the success of TikTok proves that you don’t need that much data to pretty preceding pinpoint what someone wants to watch (or what will get them to spend the most time on the app at least).

Do you mean with humans generating the prompts or with some sort of no-human-in-the-loop "generate the text prompt to generate the video" automation?

I think a super accessible animation tool would get a lot of use and result in a lot of cool stuff, but it's the latter that I'm really curious about in terms of how people interact with it.

I don’t think there’s anything making it impossible for actual intelligence to arise from a task as simple as “predicting the next token (to model human thought/speech/writing)” because with enough compute resources, smart AI implementations, and training that task basically would be optimized by becoming a general intelligence.

But it’s clear based on current implementations that once you work backwards from the knowledge that it’s “just predicting the next token” you can easily find situations in which the AI doesn’t demonstrate general intelligence. This is most obvious when it comes to math, but it’s also apparent in hallucinations and the model not being able to reason through/synthesize ideas very well, deviate from the script (instead of just answering a question with what it has already, in some cases it should not even try to answer and instead ask more clarifying questions). To be fair, there are plenty of humans with excellent writing or speaking skills that are bad at that kind of stuff too.

The problem is such an approach is limited by the content of the training texts. As I mentioned elsewhere, our written texts assume huge swathes of contextual and experiential information and knowledge that LLMs don’t have. It’s possible some of it might be inferred from the texts, but not all of it by a long shot.

If somehow you could generate a training text encoding a complete and thorough understanding of the physical world, human psychology and sociology, and reasoning then that might get you quite far. But the existing it even near future human textual corpus isn’t really that. Even then I still think you’d hit the limitations of the LLM cognitive architecture pretty hard.

simonh says >"The problem is such an approach is limited by the content of the training texts."<

Aside: I would like to see ChatGPTs with distinct training texts, e.g., a ChatGPTs trained on the "great books" of Western philosophy and science knowledge up to the time of Victorian England.

That'd be like saying that search engines are smarter than the almighty human intelligence because they know the capitals of every country while most humans don't. No, it just has access to a lot of data near-instantaneously. Just like GPT-4 does. It's the enormity of compiled human knowledge that is "smart" in GPT-4. It absolutely is "just predicting the next token", and it turns out that's enough to be an astoundingly intelligent-seeming system when trained on thousands of years of human knowledge. Of course it is! It's like in Avatar: The Last Airbender when he consults with his thousand past-lives at once for wisdom. GPT-4 lets us consult with the collective knowledge of humanity! It's absolutely amazing! And it's also "just predicting the next token". Those are both true.

There's definitely hubris in how clever we consider ourselves. And encountering these AIs will hopefully bring a healthy adjustment there. But another manifestation of our hubris is the way we over-valorize our cleverness, making us feel oh so superior to other species, for example. Emotions, desires, agency, which we share with our animal cousins (and plants maybe also), but which software systems lack, are equally important to our life experience.

> I’m no philosopher, but if your argument hinges on moving goalposts until purity is reached, and your basic assumption is that the requirements for purity are infinite, then it’s not a very useful argument.

I imagine many definitions are initially rather broad and only get refined down over time. Laertius gives us a classic example:

> Plato defined man thus: “Man is a two-footed, featherless animal,” and was much praised for the definition; so Diogenes plucked a cock and brought it into his school, and said, “This is Plato’s man.” On which account this addition was made to the definition, “With broad flat nails.”

I don’t think it’s correct to think of that as infinitely moving goalposts, however. More that the weakness of definitions isn’t always immediately transparent.

> I cannot see that it matters if a computer understands something.

I am not sure they can, but the difference is profound and material. A machine that actually understands, like a human being, is not going to be (can not be) entirely truthful or transparent. There will be private inner thoughts, idea formation, and possibly even willful intent, as a direct consequence of understanding. And the nature of interactions, regardless of superficial similarity, shifts from one of utility to relationship. For example, we would care to know if e.g. the systems entrusted with apocalyptic deterent forces are mechanisms or organisms.

Please note that not a single one of us has ever interacted with any intelligent life form lacking a sense of self, or an ego. Thus, all our sensory registers of another 'intelligent being' are learned in a context of the implicit 'this other is like me'. We are not equipped to distinguish or articulate intelligence (in the abstract) merely based on sensory information. Note that even non-verbal communication, such as jabbing a friend in the ribs, are all learned to have a certain meaning in that very same context of implicits, and any mechanism that mimicks them (via training) will be afforded the same projection of the implicit. I do not believe there is, in fact, any non-destructive test of determining 'consciousness' in an entity. (Destructive, since there may be long running tests of a subject than can be shown to be probably accurate, possibly via creating situational problems involving survival, and unexpected circumstances.)

Ask yourself what is it that convinces you that the last person you spoke with (in real life) was actually conscious? I assert that the entire matter is a 'fictional certainty' based on assumption of shared nature. "They are conscious because I am".

I'm not sure many people take the Chinese Room argument seriously these days. I mean it basically proposes a room that is equivalent to a lookup table and says it can't understand much hence computers in general can't understand. The "hence..." bit makes no sense really.

> It seems to me to posit that to understand requires that the understandee is human.

Here's a thought experiment. Suppose we make first contact tomorrow, and we meet some intelligent aliens. What are some questions you would ask them? How would you decide on their sentience or understanding?

Sentience involves goal-seeking, understanding, sensory inputs, first-personal mental states (things like pain, happiness, sadness, depression, love, etc.), a sense of what philosophers like Elizabeth Anscombe call I-ness, etc. Most of this stuff, to me, seems like is language-agnostic. Even a baby that can't speak feels pain or happiness. Even a dog feels anxiety or affection.

LLMs are a cute parlor trick, but a phantasm nonetheless.

There's no denying LLMs are anything but sentient however is sentience really needed for intelligence? I feel like if we can have machines that are X% smarter than a human could ever get for any given task, it'd be a much better outcome for us if they were not sentient.

Does that mean sentient beings need to have had the evolutionary history that people have had? After all why would aliens have feelings like we have feelings? Don't we have feelings because either it was useful to our ancestors, or they are a spandrel of something that was?

Maybe that's all there is to it. But maybe not. These questions are not settled. It might seem obvious to you, but it doesn't seem so to me. Check out some of my other comments in this thread if you'd like to understand where my reservations come from.

> if it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes.

This isn't true. If a plane flies like a bird and you only need it for flying it doesn't then follow that a plane is a bird "for all intents and purposes".

The emphasis on “for all intents and purposes” is important. If you specify requirements you specify them for your intent and for your purpose.

Requiring that something fly and that something be a plane are two different things with only minor overlap. If all you require is something that flies, then a dragonfly matches your requirements exactly as much as an apache helicopter does.

I have tried multiple times to use Chatgpt to generate Unreal c++ code. It does not do.

It spits out class names for slate objects, that inherit from other slate objects. Chatgpt doesn't understand inheritance. It just guesses what might fit inside a parameter grouping, and never suggests something with the right class type.

For my use case, it has never quacked like a duck, so to speak. It never performed, the word that might cover the concept of generating output without understanding it.

We agree on the value of computers understanding versus performing... only as much you need understanding to make it perform.

Predicting words alone does not cut the mustard, some structural depth or validating maps or some new concept is needed to sure up the wild horsepower in ChatGPT.

It must understand/have structure, or at least use a crutch to get it over the finish line..

Yes, in this instance I understand failings of today (though copilot has a much better hit rate, and at the moment it’s a great augmentation to coding if you treat it like an enthusiastic intern).

My question is about the future. The argument goes that a machine can never understand Chinese, even if it is capable of interpreting Chinese and responding to or acting on the input perfectly every time. My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding.

It’s kind of like the whole string theory vs SUSY vs… argument in physics. If the only outcomes are things that agree with the Standard Model in all measurable aspects, and don’t provide any measurable distinction, then for all intents and purposes they don’t matter. That’s why their active areas of research are looking for the measurable distinctions.

FWIW, supersymmetry models predict measurable things (that so far have only ruled out those models when tested) but have applications elsewhere. String theory research has had implications in mathematics, condensed matter, and a bunch of other places. They’re useful.

But that’s beside the point, because the premise of the Chinese room problem is that there exists a machine that passes all scenarios, where no measurable difference can be found, and that this machine does not understand Chinese.

> My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding.

I'm not sure if you understood the argument. The argument isn't asserting that there is a measurable way of distinguishing it, it's actually claiming that regardless of how well it seems like it understands Chinese, it doesn't actually understand Chinese. It's about intentionality and consciousness.

The argument is that the man inside the room doesn't understand Chinese. The entire system still obviously and demonstrably does.

In a chatbot, the man inside the room is the LLM, but the whole system is not just the LLM - it's the whole setup that picks generated tokens and feeds them back into the input as a loop. And it demonstrably understands what you tell it, because it can carry out instructions, even extremely convoluted ones or using substitute words that are not part of its training set.

You are entirely correct: I do not understand the argument - or, rather, the relevance of it in any applied situation.

In effect, my argument is that in order for you to require it to understand something, you require it to understand that thing for a reason. If it acts like it understands that thing under all probing, then your requirement is satisfied - the question about whether it truly understands the thing is moot, because it fulfils the requirement.

It's hard to keep this theoretical. Yes a machine is just a machine.

Defining a machine to be conscious, allows the individual to soak their mind in code and silicon as a receptacle for their spirit.

It creates a pull into a 'second mind'. Anybody who believes this is likely to invest heavily in the maintenance of new technology.

A 'conscious machine', creates an uneasy feeling that we should work to embed our spirit, knowledge, intellect into flipped bits, like expectant mothers. That we should work for the machine, and to the ends of the machine.

And that machine is somehow defined-to-be or a naturally, consciously alive (to a large or small degree). It is said to have a mind worthy of a person's professional output and it can hold the power of a marginally believable conversation.

While all of these described properties are vaguely plausible, it does nothing to help me understand the meaning of a technology, and only benefits those looking to create a fevor around a new tech product.

Describing chatgpt as a stochastic parrot or chinese room grants me a metaphor or analogy for the inner functions of the tech. It also lets me see, or otherwise guesstimate the products abilities clearly, without the belief-as-marketing hype.

I can take the stochastic parrot metaphor, to an article about LLMs and understand in a couple of days what took years of research to create.

Following the belief of computing as real human intelligence and that human intelligence is fundamentally mathematical, requires on some level submission of your mind to a machine that has it's own goals programmed in by someone else.

This centuries-long process of trying to encode and store all human knowledge behind the secure walls of complex coded signs.. and it's advocates for that process, create a subtle and deep twinge of future melancholy or dread or something. The idea that all written/typed meaning will be accessible only by the spiritual power brokers, and not our sons.

No. On some level, machines are just machines, like an abacus or a weaving loom. It can host concepts in the same way that a weaving loom is 'intelligent'. It holds it's shape, abstractions and functions by the laws of physics/metaphysics and according to my human dictates.

You follow the raven into the computers-are-conscious dream at your own risk. Computers are leaning towards controlling people rather than emancipating them. Leaning very hard in that direction. Do we want that? Freedom of mind and meaning is valueable.

Yeah, I asked it to write some relatively simple Zig and it produced a weird mix of Zig and C. It just made stuff up, fake it until you make it style.

I don't know if I take the Chinese Room scenario seriously, it's a little hard to think about. For me the key is that an LLM (and maybe the Chinese room?) is only trained on text, and therefore its entire "universe" is merely representational. To the LLM there is no difference between the color blue and the word "blue", or a dog and the word "dog". People understand that a word is a provisional marker for something with almost infinite complexity, and when a word fails to meet the goals of the person, the word is discarded, amended, or elaborated until it expresses the thing which has heretofore been unexpressed. For an LLM there is no deeper world to access beyond text.

I don't think an intelligence needs to be human, and it should be physically possible to create an intelligence which is synthetic. But in order to call the intelligence "general", and to rely on it for the purposes that designation implies, it would need to be able to successfully navigate the world, which requires access to that world and the use of the world as its own model, rather than the much simpler and coarser intermediary of text. In order to claim that an LLM can fully navigate the world after being trained on pure text, we would have to believe that all our writings across history have exhausted what there is to say about the world. This is not to say an LLM cannot be useful for some purposes, but there will be key ways in which they fail because they have no sense of meaning or what the world is like. Whether consciousness is required to solve this I don't know, but we simply haven't begun to approach a system that can meaningfully address the world as a world.

Here's GPT output that demonstrates that it doesn't reflect on its text: «GPT Chat: You are right, I apologize for the confusion in my previous responses. The corrected explanation is:

The pattern is counting the number of closed spaces in each letter of the spelled-out number. A closed space is any enclosed space in a letter, such as in the letters "a", "b", "d", "e", etc.

Following the pattern:

- one -> 2 (there are closed spaces in the letters "n" and "e") - two -> 1 (there is a closed space in the letter "o") - three -> 2 (there are closed spaces in the letters "h" and "e") - four -> 1 (there is a closed space in the letter "o") - five -> 1 (there is a closed space in the letter "e") - six -> 0 (there are no closed spaces in the letters) - seven -> 2 (there are closed spaces in the letters "e" and "n") - eight -> 1 (there is a closed space in the letter "g") - nine -> 1 (there is a closed space in the letter "e") - ten -> 1 (there is a closed space in the letter "b") - eleven -> 3 (there are closed spaces in the letters "e", "l", and "v") - twelve -> 2 (there are closed spaces in the letters "b" and "d") - thirteen -> 2 (there are closed spaces in the letters "b" and "d")

Each item follows the pattern, as the number of closed spaces in their letters matches the corresponding number in the pattern.

The whole sequence is:

one -> 2 two -> 1 three -> 2 four -> 1 five -> 1 six -> 0 seven -> 2 eight -> 1 nine -> 1 ten -> 1 eleven -> 3 twelve -> 2 thirteen -> 2 ...»

It is clear the model doesn't know what it is talking about.

Anything having to do with shapes or spatial relationships seems to befuddle GPT-3.5. GPT-4 is significantly better but it's clear that it still struggles with it. As far as shapes of the letters in particular go, you can ask it to draw them using SVG or <canvas> and see what comes out.

Here's a different example involving dataset analysis with GPT-4 that required it to analyze its own previous outputs to find and correct mistakes and form a new hypothesis:

https://gist.github.com/int19h/cd1d1598f91e8ba92dd8e80bd5d21...

Isn't the whole point of Searle raising the Chinese Rooom argument exactly so that these kinds of things can be debated?

This is also my interpretation of the Chinese Room.

Norvig and Chomsky really got into this type of argument, though maybe it’s a stretch to say it’s this exact one; see Norvig’s side here: https://norvig.com/chomsky.html

I literally lost a friend of thirty years yesterday because she is wedded to the Chinese Room analogy so fiercely, she refuses to engage on the subject at all.

For all the terrible things people worry about ChatGPT doing, this was not one that I thought I was going to have to deal with.

(edit: ChatGPT was not involved at all, but when I suggested she give it a try to see for herself, that was the end of it.)

How can an disagreement over GPT possibly be something to end a friendship over?

This sounds like you said "I cannot possibly be friends with someone who does not believe that LLMs are emerging AGI!", and people read it like that and are downvoting you.

I'm gonna assume the situation was more complex, but still find it hard to imagine, how a disagreement over such an academic topic could end up destroying a friendship.

It was a complex situation to be sure.

I only shared the story to illustrate how personally people are taking these discussions. I really felt like I was being very neutral and just sharing my enthusiasm. It was entirely unwelcome, apparently.

If there's a lesson to be learned it's that people's tempers over these issues may be hotter than they appear.

I can barely speak with my artist friends on the issue these days due to their generative AI fears. Their emotions are completely intractable on the subject: AI art is theft. Period.

If are is a major part of someone's identity (strongly implied by calling them an "artist"), then it's hardly surprising that they would feel threatened and react emotionally to something that massively disrupts the creation of art and the viability of making a living from it.

I'm old enough to have had my design and publishing career evaporate underneath my feet because newsprint has mostly been replaced by the internet. I'm even old enough that I was part of replacing other people's careers with technology during the desktop publishing revolution. I remember when graphics tablets became a thing and how traditional pen and brush artists cried, "unfair!"

I always felt empathy for them, even back then. I feel empathy for my artist friends as well.

How can we ever have a better future if we keep trying to prevent change from disrupting our careers? In my experience, I've seen a lot of wailing and gnashing of teeth, but these changes have always come no matter how hard people fought. On the other hand, I see people who try to adapt to these changes do quite well.

It's the part where I watch them hide their head in the sand that bothers me, not their anger. I want them to succeed. I think they have great eyes for color and line and those things are still super-valid long after the generative AI has replaced the pen and brush.

I don't want to watch my artist friends suffer. I am old enough to know that the effort to stop generative AI is almost certainly going to fail (just as all the previous efforts to prevent change failed), and it is better to light a candle than to curse the darkness.

The only constant is change.

JeJie says >"Their emotions are completely intractable on the subject: AI art is theft. Period."<

Then the only true artist would be one who has never seen art before except his/her own. The correction of your friends' belief would be that "ALL art is theft. Period."

You blew up a 30 year friendship over an...analogy?

I didn't! Someone else did it to me. I was trying desperately not to.

(edit: This is the kind of stuff I think my friends are watching and being informed by [0] as it was what they are posting in our common areas.)

[0]: https://youtu.be/ro130m-f_yk

Why does it upset you if they disagree with you on this?

It's not that they disagree with me, as there were many things we agreed on. I have read over it and it seems that they don't respect what I believe so much that their example of how they felt was this angry shouting man. But again, I was not the person who ended the friendship.

In short, I'm upset that a person I thought was a friend disrespected my opinion about something so much, she stopped being my friend.

You're talking about intelligence - GPT doesn't quack at all. It clearly is not a duck.

Reply on news.ycombinator.com