Comment by goodside | Hacker News Reader

Comment by goodside | Hacker News Reader

Parent Full thread

Having spent months talking to GPT-3 on a daily basis, I assure you it is not conscious. It has no perception of time, no awareness of what date it is, no memory of past experiences. It doesn’t want anything — it has no goals. The question “Is it conscious?” isn’t interesting to anyone who spends time with it.

I will say, though, that it bugs me when people say it can’t be conscious because it sometimes says stupid things. In most cases, there are known tricks to suppress undesirable behaviors. More to the point, though, if we encountered a human being who gave confabulated answers to questions like “When was the Golden Gate Bridge transported for the second time across Egypt?”, we wouldn’t insist that this human is not conscious — we would just call them brain-damaged or mentally ill. I don’t think modern models are conscious, but as a logical possibility they could be conscious and still say very stupid things all the time.

It’s a static model, it does not learn after the initial training. That alone rules out any possibility for consciousness imho

It should have a feedback loop from the conversations so that it could keep learning. Maybe by users rating each response.

It could learn things via the context window. Maybe there'd be no long-term conscious state but you could keep short-term coherence well enough. I think the real problem is that it simply isn't capable of training for self-awareness and reflection because there's only very indirect evidence of it in the training set. We talk about Chains of Thought, and it's a very powerful technique, but this is all just the tiny snippets of text on the internet where people happened to narrate their thought process out loud. If it could learn on self-generated instances of using Chain of Thought to help it predict other text in its training data, this tiny fragmentary piece of consciousness would get massively reinforced.

Exactly - tt has no memory except the narrow window. When these models develop a hippocampus analog and can push things into long term memory, it might become interesting.

I think I heard of people with brain damage who does not keep an internal state, they feel like they continuously wake up and only remember memories from before the brain damage. That would be a kind of parallel to consider, or at least somewhat analogous. I think most of us assume that such people would have phenomenal consciousness / subjective experience / qualia

I don't think that's a fair comparison. You can't put a biological system into 'read-only' mode like you can with a computer system (where this is the default since modification requires dedicated effort).

A biological system self-modifies even when just associating memories. The phenomenon you describe is much more high level, we observe someone can't form new memories, but that doesn't mean their fundamental brain chemistry has changed.

Then the question is whether it was conscious during training

It's interesting how people who've used GPT-3 seem to be learning how to use it better. That leads to the question of whether you could get GPT-3 to do a task that involves tailoring a query to itself. Ever tried that?

That only happens when the prompt is constrained to force an answer even if a nonsense question is being asked. If you allow an out, the model knows to refuse answering when it detects a nonsense question.

Reply on news.ycombinator.com