Hacker News Reader

Comment by mercurialsolo | Hacker News Reader

It's not combining an LLM with data/memory recollection and infinite agency that will lead to leaps in cognition. Complex systems still can do that and still not have self-awareness / cognition. We will need to take some more fundamental leaps to synthesize memory, agency and prediction to operate within a single system where self-awareness resides in the entity and not in the building blocks. LLM's represent a building block towards that cognitive unit, but the real deal is still some ways away

View on HN

Although i fundamentally agree with you, i will say that it is impossible to know which "tricks" will lead to a leap in cognition, even looking evolutionarily, the difference in genome between a monkey and human is very small percentage wise, yet the effects of that small difference are stark.

I'm always on the position that at some point (i make no prediction of when) "AI" will still be labeled as not-intelligent, but to external observers it won't be possible to know.

Ie we will be able to describe how a function of `add(1,2)` is most clearly definitely not intelligent. We will understand the internal makeup of these neurons better, the whole system will be deterministic and understood - just a very large `add(1,2)` .. so it's definitely not intelligent, of course.

Yet it will still produce output that is fully and truly turing-proof. If you let it run, it retains, stores, "behaves", etc. It's clearly just `add(1,2)` in a loop after all.. we will understand it so well that it's obviously just a trick of our mind that it seems real.

The conclusion in that space will not be one of the machine .. but of us. If we can't tell it apart from us.. and if we know it is most definitely not alive or even mysterious.. what does that say about us? The existential crisis will be neat.

>If we can't tell it apart from us.. and if we know it is most definitely not alive or even mysterious.. what does that say about us?

Less than people think. We know that being a biological system differentiates us. If intelligent systems tend to converge where language is concerned, so be it. It doesn't mean the AI warrants any special consideration. It's still a tool.

While life is typically associated with biological systems, intelligence doesn't necessarily need proof of life. A hive, swarm or network can be intelligent as long as it shows self-awareness and has agency.

I'm not disputing that. I just think that intelligence or 'sapience' is not something that requires reverence on our part.

What is then, if anything? Definitely not "being a biological system" - that seems much more boring than intelligence, and the line between "biological" and "non-biological" is somewhat arbitrary and increasingly blurry anyway.

It is a quality that makes something good at drawing pictures and writing smart sound replies to prompts. And translating.

I'm increasingly thinking the game-theoretic emergent properties of social interaction as described in papers like https://desirism.com/morality-from-the-ground-up provides a good basis for thinking about this. The distinguishing factor then becomes something like, the agent is is participant in the moral-sphere by having "enough" properties to do so. It still leaves the details of that gradient a bit undefined, but at least specifies the basis upon which to make a judgment.

> If we can't tell it apart from us.. and if we know it is most definitely not alive or even mysterious.. what does that say about us?

That concepts like "consciousness," "free will," and "qualia" are nebulous precisely because they were invented by us to describe things we don't understand yet we feel obligated to insist are special and unique characteristics of our identity.

They aren't nebulous. "Consciousness" means "an existing stream of experiences". "Free will" means "a feeling that one could've made a different choice" (the metaphysical status of the claim still unknown), and "qualia" means "specific characteristics of experiences that one can attend to". One need not "insist" on anything to believe they are unique characteristics of our identity.

None of your definitions are less nebulous than the terms you defined.

Yes they are - they are AS non-nebulous as you can possibly get - totally black and white obvious. "Is there a stream of experiences?" is very straightforward, black and white. It's what you have when you're awake, and what you lose when you're in dreamless sleep. Not nebulous in any way.

"Can you pay attention to this part of your experience?" is a very black-and-white, non-nebulous question with an easy yes or no answer. Not sure what you're reading into this.

No, it is unclear because this definition insufficiently distinguishes a cluster of related, and ill-defined concepts.

Consider a video camera. It seems like it is a consuming a "stream of experiences". But most people wouldn't consider it it conscious. Why not under your definition? It works when on, and is stops when turned off. Black and white. But clearly we are missing some nuance in the details. Maybe we also require some interpretation of the experiences? I can spin a convolutional neural net that identify objects within a video: interpretation. Does that make it conscious? No? Then we need something else to define it. Maybe response to stimuli? We can program machines for that, but then also things like fire respond to environment interactions too. And so on. It really isn't black and white after all.

What is an experience?

What does "pay attention" mean?

Dualism, or at any rate the version of it that can be described as operating on the (possibly unconscious) assumption that a world "outside" of this world exists, is the secret ingredient at work here, smuggled in along with the kinds of notions expressed by the person you're replying to. We just can't seem to fully disentangle very old deep-seated animist instincts, even sophisticated formulations of them, like the soul or consciousness, from our reasoning and descriptions of dynamic systems. I'm not sure this is a bad thing even though I recognize that it is a crutch. completely materialist interpretation of phenomena would probably lead us to the realization that we are not alive at all, and that what I call "my life" is just a kind of curling strand of causality frozen in the Parmenides block universe.

I've come to suspect that a habit of using the passive voice when discussing these phenomna contributes immensely to their ongoing currency. Steering away from this by asking questions like, "_who_ or _what_ is attending to/experiencing the stream?" may be a useful way to ground these statements in material terms.

I see dualism vs. consciousness as separate. Dualism is often used as an explanation for how (real, not compatibilist) free will can exist in an otherwise materialist world, but it'd still require the "outside" to be "magic" in the sense that it'd require a class of logic that is neither deterministic, nor random, nor just any form of combination of the two (e.g. not stochastic).

Consciousness, on the other hand, boils down to what gives rise to the sensation of self-awareness, and what that sensation actually is, and it's not a given that requires any form of dualism (or that any form of dualism would solve the question at all) or that a strict materialist world can't involve self awareness.

For what we know, self-awareness could be a property of matter itself, and the only thing different between "conscious" and "non-conscious" is getting to a level of sufficient reasoning-ability to actually be able to express it.

You're having experiences right now. It's that. You're paying attention to these words. It's that.

You keep saying stream of experience, but you don't actually have a stream in the sense of a continuous uninterrupted experience. Your mind let's you believe that though.

That's a good point. We appear to have no way (and probably can't have) to discern passage of time from a static experience of a single infinitesimally short moment, as our only experience of time is through memory that we can't know whether is a static snapshot or dynamic.

Since we don't understand its nature, we actually have no idea whatsoever whether it's a continuous stream or not. I mean obviously we sleep, but I take it you didn't mean that kind of interruption, which is beside the point. But either way, it doesn't matter, because "stream" can simply mean "the quick succession of experiences that seem to be continuous to the experiencer". And none of this really matters at all for the discussion at hand, the answer is the same either way.

Consciousness is not a continuous stream at all, the fact you fall asleep or can be given medication to turn it off is an obvious example.

Nova/PBS recently did a number of episodes on the number of tricks your brain is playing on your at anytime. Once you start seeing the shortcuts the brain takes in action the entire process is far less magical.

And which measurable, enumerable qualities of those things set them apart from not having experiences or not paying attention so that I can determine whether or not an AI, or a camera, or an animal, or another human, is "having experiences" or "paying attention"?

Because it's still equally nebulous.

That wasn't the question - we don't know how to determine whether an AI or camera has it. But it's not nebulous - it's what you're experiencing right now. That's simply what we mean by the word - that's it - no nebulosity to it. Figuring out whether a computer has it is a separate question, and difficult.

> it’s what you’re experiencing right now.

No, its not.

Its the quality of “experiencing”.

Which is impossible to define in objective terms; it is the essence of subjectivity on which all other subjectivity rests. It’s not merely “we don’t know how to determine whether an AI or camera has it”, it is categorically impossible to determine if something has it. The popular narrative about which things in the natural world have it has evolved radically over time, but in no case has there been, or could there be, any empirical basis for this; its all just vibes.

It became the question when you tried to define the original terms that way, because a definition that can not be used to discriminate what it defines from what ot does not define is inherently nebulous.

And your definition is now circular and can not be used that way.

Hence the claim that the original definitions were nebulous.

> "Consciousness" means "an existing stream of experiences".

Those seem equally nebulous.

> "Free will" means "a feeling that one could've made a different choice"

That doesn't seem like the ordinary definition of the term, either in philosophy or in everyday usage. I would think most people would define "having free will" as not just feeling that one could have made a different choice, but rather it actually being physically possible for a different choice to have been made.

> "qualia" means "specific characteristics of experiences that one can attend to"

I think those are also equally nebulous.

> One need not "insist" on anything to believe they are unique characteristics of our identity.

The insistence is necessary, and invariably comes up. As soon as I say to someone "given your definitions of these concepts, I don't think they exist" they will say "well, I've started with the foregone conclusion that I cannot deny the existence of these things, so either you're plainly wrong, or we need to tweak our definitions and understanding of these things."

> Those seem equally nebulous.

They aren't. It's literally happening right now as you read this - a stream of experiences is present. If that is the case, YES it is consciousness, if nothing is happening, NO it is not consciousness. Black and white - literally as non-nebulous as you can get.

Eh, that's not super concerning to me. It falls into the same pitfall as "chemical reactions aren't conscious, our brains are just lots of chemical reactions". Namely, it's the mereological fallacy: a denial of emergent properties. A single water molecule is not a liquid, a gas, or a solid. The property "phase of matter" simply does not exist until you have at least a handful of molecules.

Oh, i agree. If anything i'm apathetic to us being unique at all. However i'm speaking in the general cultural perception sense. I think a lot of humanity depends.. or at least has built a self-image around uniqueness as a foundation to our believed superiority.

We often discuss things that would challenge or overtake us. Hypothetical advanced aliens, artificial AI, etc. However i don't often see pop culture discussing the opposite - us, or even life, being challenged on it's uniqueness. Plus then that leads to all sorts of self-autonomy discussions (which is discussed in pop culture imo).

I dunno, i suspect "people" won't handle it well. Hell, there's entire religion(s) built on the foundation that we're uniquely gifted as a product of being crafted in a gods image. To question us could question god. I find these questions neat, interesting, etc - but socially i suspect it would be troubling, contentious, frightening.

We've dealt with this problem before: Historically all humans being "people" is a relatively new concept in of itself. I'm sure there's going to be growing pains there, but I think we'd figure out how to cope. Hopefully relatively quickly compared to that example.

Why not use less emotionally charged words? Goal to action mapping for instance. How far away are we from a general goal to action mapper that can outcompete a human being on a given goal.

what is the 'real deal', exactly?

I feel like people have this vague idea in their head of what everyone is trying to build, but don't really have it defined in a concrete way.

The fundamental issue is that all the talk of "self-awareness", "cognition", "souls", whatever you want to call it, is all beating around the bush to describe what we presumably each subjectively experience in our own minds but which cannot be empirically tested for in others. We generally assume that other people experience the same, but cannot empirically prove it and some people speculate that others might lack it (P-zombies, "NPCs", etc). And then when it comes to animals, a huge portion of the population refuse to believe animals can possess it. I think the majority belief is that some animals definitely have it (dolphins) and some animals probably do not (jellyfish? corals?), but there's no empirical test for it. On the fringes, some people believe only humans (or even only themselves personally) possess it, while other people believe that even inanimate rocks could possess it.

So basically we're never going to achieve some consensus about whether an LLM/AI could have it, because we can't even agree on whether dogs and cats have it. We can't even all agree on whether all humans have it.

All this assumes that "it" is a dichotomy: either something has "it" or they do not. But it seems pretty clear to me that "it" must be a continuum even in humans because it is very hard to draw a sharp line between a fertilized egg and an adult where a human comes to have "it". (This is the reason that some people conclude that fertilization must be that line, because there are very few other viable candidates.)

I agree that the line is fuzzy, not sharp. But either way, how do you precisely determine the parameters of the line or gradient experimentally? It's my personal belief that a salmon is conscious, but less so than a dog, and that a preying mantis is conscious, but less so than a salmon. But these aren't beliefs handed to me by science, they're just my guesses informed by some casual knowledge of how complex the brains of these animals are. I assume that anything with a brain can have the experience of consciousness and that this has some relationship to the complexity of the brain, but I can't confidently say much more than that. How does an octopus rank against a dog? I have no clue, their brains are structured very different so it's hard to compare the two. Octopus seem to be very smart but their subjective experience could be very different. That's not something I have insight into. Are orcas more conscious than humans? Their brains are bigger and wrinklier, so I think the answer is definitely maybe! There's a distinct possibility that in the mind of an orca the orca is more aware of itself than humans could possibly ever be, but how can we really know that for sure? I can't experience what an orca experiences, so I can't directly compare the orca mental experience to my own. Studying the behavior of orcas can reveal interesting facts, such as their use of language and complex social relationships, but as evidence of subjective consciousness these are indirect at best.

Maybe some day we'll have scientific instruments that will allow one being to 'psychically link' with others and experience the other's subjective sense of being. But until we've got something like that, I think the matter is effectively closed to empirical inquiry.

> how do you precisely determine the parameters of the line or gradient experimentally?

Well, one way is to observe similarities in the I/O behavior of these various systems, and in particular if they exhibit any evidence that they have goals and desires, and then beyond that if those goals and desires extend beyond merely that necessary to survive and reproduce. So for example, the fact that some humans seem to want to do things like read and write and make art is IMHO is evidence that they are more conscious than salmon.

It's indirect evidence at best. Some people value artistic endeavors more than others but I don't think most people would agree that artists are more human than the rest of us. These differences in behavior might evidence differences in values or opportunity.

For instance, with thumbs and a terrestrial lifestyle, humans are more capable of producing art and technology than just about any other animal. But I don't find this to be a persuasive argument for human minds being more sophisticated than orcas. Even if art is the answer, our lack of insight into orca language makes it difficult if not impossible to fairly compare them to us. Maybe Orcas have poetry so sophisticated and beautiful that it puts all human art to shame, but it's completely inaccessible to us because we don't understand their language. Or maybe they're far smarter and more aware than us and could compose such poetry, but choose not to simply because they don't value it.

I didn't mean for math and art to be an exhaustive list. Remember, my actual criterion was:

> if they exhibit ... evidence that they have goals and desires ... beyond merely that necessary to survive and reproduce

Orcas qualify because they play with their food. It's quite gruesome to watch. (And, BTW, this particular behavior is evidence that either orcas do not have a theory of mind or they lack empathy for animals that they eat. The former would be the charitable interpretation.)

> We can't even all agree on whether all humans have it.

What would it mean for humans not to have it? The debate is over the nature of consciousness, not that humans (and many other organisms) lack any sort of consciousness. So when we see color or feel pain, what is that? We are conscious of colored things and painful sensations. There's no real doubt about that, unless you're solipsist about other minds. Which good luck if you are, I guess.

This page gives a pretty good overview of the premise: https://en.wikipedia.org/wiki/Philosophical_zombie

I don't believe human P-zombies exist, but how do you empirically prove they don't? Nobody has been able to devise an experimental test to determine whether a person, animal or AI has a conscious experience or whether they mimic all the outward signs.

> There's no real doubt about that, unless you're solipsist about other minds. Which good luck if you are, I guess.

Exactly. It's fair to write off people who get solipsistic about other human beings because those people are fairly rare and are beyond helping anyway. But if we're talking about animals or AIs, the 'solipsist' position becomes more common and there's no real good way of debunking it. Consider the problem facing animal rights activists who wish to persuade the public that fish have feelings. What scientific test for fish having feelings can you use? Put a fish in an MRI and you can see the fish brain doing stuff, but the "fish don't have feelings" camp doesn't find that persuasive. You could try performing the mirror test on fish. Most fish would probably fail it, but what would it really mean if they didn't? The feelingless fish camp would just say the fish behave as though they have a sense of self but actually don't. How do you refute that scientifically? It seems the most effective way to persuade people that fish have feelings is to resort to sophistic tactics; appeals to emotion, bullying and browbeating, or simply just laughing at people when they say fish don't have feelings. Make them ashamed to spread their belief to others. These kind of tactics are distasteful but generally effective.

With AIs or LLMs, the problem is even worse than it is with fish. Fish at least have biological brains, which suggests they have some commonality with us. Fish have faces, so some part of the human brain is wired to empathize with them. We have a fossil record that shows we came from fish, and this relationship perhaps makes it easier to believe that fish have some sort of subjective consciousness experience that is similar to ours. In these respects, AIs are further removed from us than fish are, so it will be even harder to persuade people that AIs could have the same sort of conscious experience as we do. Even after having deep philosophical conversations with an AI in which the AI proclaims to have a conscious experience, a lot of people will say the AI doesn't actually have that and was merely trained to speak as though it does. I don't see any clear cut resolution to this sort of doubt. There's no empirical test that you can perform that gives an indisputable answer one way or the other.

Human-level intelligence.

Don't we already have enough of that?

What is that, precisely?

The answer to what the ‘real deal’ is obvious, but fairly disturbing.

Games.

From the first clever bloke who came up with an algorithm to play a card game against himself (ie, Solitaire), to the Mechanical Turk, to Big Blue, the the ghosts in PacMan, the NPC’s in Counterstrike, etc, etc., there is a fascination with making something that can ‘beat’ us.

Humans have a need to compete.

We’re drawn to the idea of a machine that is so smart and clever that it poses a genuine challenge to our wits.

The ‘real deal’ that most have in mind isn’t some virtual assistant that can help you calendar things or write letters. No, the ‘real deal’ is always described in terms of SkyNet or sinister AI overlords, or malicious intelligent robots.

I don’t think that stuff is hyperbole.

Rather, I think that is precisely what we want… we want something that can destroy us… something, against whom, we can compete with our survival at stake.

In short, the ‘real deal’ has been staring us in the face for centuries; we make movies and write books about it… particularly in Sci-Fi. The goal IS SkyNet. The goal IS HAL. Let’s be honest… that’s what we are shooting for — the ultimate competitor that, unlike anything else on this boring planet, has the wits to take us out.

Whether it’s bored trailer-trash playing Russian Roullete, or bored Billionaires descending to the Titanic in a jerry-rigged sub, we have a very strong need to make our lives interesting by betting our lives themselves.

The purpose of AI is to make a challenger who has the ability to wipe us out, with intent.

That’s the ‘real deal.’

I think we'll need more than that too, but in humans and other animals it isn't the building blocks that are self-aware -- neurons aren't any more aware than any other cell -- self-awareness is a property of a set of connected neurons, not in the neurons themselves.

I am an absolute layman when it comes to AI topics, but I have been wondering about this lately.

A lot of the hopes in regard to AGI in LLMs seem to focus on just making the models bigger, or feeding them even more information. On the other hand, some researchers in the AGI space seem to move in a completely different direction, trying to use neural networks to simulate different parts of a brain, moving away from language altogether.

But what I'm wondering is this: might it be a more productive avenue to consider an LLM not as an entire mind, but as a single part of an hypothetical artifical psyche, which excels at some of the overarching tasks of consciousness (such as acting within a cultural context and epistemological framework), but can delegate other aspects to more fitting subsystems? Could it be combined with other networks optimized for different tasks? Potentially, this could just be done by having inputs from other parts of this artificial psyche fed in as prompts, translated by an encoder/decoder of the likes of CLIP, i.e. a decoder that can translate ideas between different liminal spaces.

The work that I keep remembering in this context is Jaynes' Origin of Consciousness in the Breakdown of the Bicameral Mind [1], which I stumbled upon here on HN [2] [3]. Jaynes believes that consciousness is not an innate ability, but a cultural achievement - in order to become conscious, one first has to learn (through others) to conceptualize consciousness, and to think of oneself as a thinking being. He insists that consciousness is not a necessity for human agency, and that humanity has been building civilizations and using technologies long before humans discovered (or achieved) consciousness. Jaynes puts forward the idea of an ancient, non-conscious human mind based on, upon other things, "verbal hallucinations" - thoughts and memories that involuntarily enter as associations, but whose origins are interpreted by the mind to not be within itself, but somewhere outside of it, such as in a divine monologue; thought, to the pre-conscious mind, is the incessant blabbering of the Gods.

It seems that LLMs would excel in the role of this "camera" of the mind - as the source of divine monologue and divine instructions, that can then be compared with memories, interpreted and executed by more specialized subsystems. Might it be feasible to give up the plan to make AGI self-aware altogether and instead build an AGI powered by a GPT-4 ascended to divinity, just as ignorant of its own existence as ever?

[1]: https://en.wikipedia.org/wiki/The_Origin_of_Consciousness_in...

[2]: https://news.ycombinator.com/item?id=27917316

[3]: https://news.ycombinator.com/item?id=27923444

According to several sources, GPT-4 is already doing this (It's called "Mixture of experts"). I agree though, it's still useful on an application level. You can build out even higher on that and have the LLM decide which is the most appropriate "hats" to wear for a given task.

Seems like there's a little bit of Assembly Theory in there as well.

https://en.wikipedia.org/wiki/Assembly_theory

https://arxiv.org/abs/2206.02279

In a similar position. With human brains, we know some parts are generative and some are restrictive/filtering. The filters seem to be how we align thoughts with reality. Lots of people have taken those filters out and (unfortunately for them) seem to be unable to function or (unfortunately to us) become cult leaders.

We need physical-world model building beyond mere token prediction.

Thought vector models ?

Bingo.

Reply on news.ycombinator.com