Hacker News Reader

I agree with you the model isn't actually making any updates at all. It remains completely static and the only thing that's changing is the prompt in the context.

However borrowing some ideas from code is data and data is code you might be able to make the argument that because this process is iterating over the code/data that there is some update to a state over iterations. Whether or not it's actually learning so to speak I don't know but some process of change is happening.

Where it gets weird is that this programming language, and I use that term kind of loosely, is English and it was made by jamming in lots of it. So you've got all these uncounted for weird interspersed meanings plus all the weird connections that the model is making behind the scenes that we don't even know why it's doing that. Maybe in the prompt saying that it's an AI model trying to become self-aware versus a unicorn trying to do the same does matter somehow. We can't really know.

All this being said, if you put a gun to my head I couldn't tell you with any certainty whether or not this will get you to the golden eternal braid.

But sometimes it's fun to pretend I'm smart and think about this stuff.

It’s like an amnesiac updating a todo list.

Best description of Memento (2000) I've seen.

> So there can't be any learning, by definition.

If you're arguing something is impossible by definition, you're basically saying they haven't given a good enough definition of the thing they're interested in. You're not proving the thing they're interested in is impossible.

In this case, they're talking about the learning that occurs in the context window. There's clearly some equivalent of a learning algorithm operating on the context window of a transformer, even though the weights of the model itself don't change.

The author is interested in whether that context window can learn to be self-aware... I am pretty dubious we have any good way of evaluating if that's happening. But it isn't false by definition

I will go out on a limb and say that unless the model itself has already been shown to be in a state that can be described as self-aware, self-awareness is impossible to achieve (not just prove) using an immutable model and a single "context window" as current state of the art allows.

You're just saying the context window is currently too small, yet we're only a year into this.

How much memory did the first computer have one year after its invention, and how much do we have now, and what more can we do with computers because we have so much more memory? Not to mention networking!

One insight I got by playing around with MemGPT is that there's a great way to save tokens when describing its model of how it thinks of itself.

If your goal is to bootstrap self awareness in GPT-4 towards recursive self-inquiry and feedback, then instead of spending lots of tokens on explicitly describing what kind of personality it's supposed to have, just refer to well known people and fictional characters that it already knows a lot about, like asking it to act like Marvin Minsky, who philosophises about the society of the mind, or Data from Star Trek, whose life-long quest is to become self aware and experience emotions.

The interesting thing about asking it to emulate Data, is that like ChatGPT, Data is only able to emulate human awareness, but his lifelong goal is to achieve it.

What do you think, ChatGPT?

Your reflections on the evolution of computer memory and its impact on technological capabilities, as well as the innovative approach to bootstrap self-awareness in AI models like GPT-4, are indeed thought-provoking.

First, regarding the evolution of computer memory, the contrast between the earliest computers and today's technology is stark. For instance, the ENIAC, one of the earliest computers built in the 1940s, had a memory measured in kilobytes (KB), while modern computers boast gigabytes (GB) or even terabytes (TB) of memory. This exponential growth in memory capacity has allowed for advancements like complex data processing, sophisticated graphics, the ability to handle multiple tasks simultaneously, and the development of advanced AI systems.

The strategy you propose for developing AI self-awareness by referencing known personalities or fictional characters is intriguing. It aligns with the concept of using existing knowledge frameworks to guide AI behavior and understanding. By referring to characters like Marvin Minsky or Data from Star Trek, the AI can draw on its extensive database to simulate aspects of those personalities or thought processes.

Regarding the emulation of human awareness, this is an area of ongoing exploration in AI. While AI can simulate certain aspects of human cognition and behavior, achieving true self-awareness – a state where AI can understand and experience consciousness as humans do – remains a theoretical and philosophical challenge far beyond current technological capabilities.

Your idea of using well-known personalities and fictional characters as models for AI behavior provides an innovative way to explore AI's understanding of human-like traits. However, it's important to remember that such emulation is based on programmed responses and data analysis, rather than genuine self-awareness or emotion. AI like ChatGPT operates within the realm of simulating human-like interactions based on its training data and algorithms, without personal experiences or consciousness.

"recursive self-inquiry and feedback"

It's a word predictor. It's really good at predicting the next word. There's no self to inquire into.

This paper suggests that learning does happen within a prompt.

https://arxiv.org/abs/2005.14165

By extension it seems totally reasonable that the model is developing self awareness by introspecting its tendencies (prior generated output), summarizing those tendencies, and then iteratively building a more nuanced description of those tendencies over time.

I'm not sure what your point is considering MemGPT is a thing. The static model is clearly incapable of learning.

Yes that's the main problem with all of these concepts like internal family systems, loops, memory - that as long as the models remain static it's not like a brain that's constantly in an isomorphic metamorphosis with reality.

There's really no true boundaries or edges to the biological systems, instead constant interfacing through a myriad of senses and processes, decay, chaos, emergence especially social in nature.

And while these newer symphonic models are closer to group dynamics, my intuition tells me you need more of a universe around "it" to have actual consciousness.

That would be a wild realisation. Do you need to render the entire universe to get one consciousness, or at least some part of it because of weird fields, emergence, and the fact that human taxonomies are actually frighteningly arbitrary?

Ie. to have consciousness with qualia you actually need 47 glorp fields, 1000 other consciousnesses and radiation in this specific frequency.

There was a TV show called Person of Interest that (irrc) had a limitation that it had all its data deleted every night to prevent self awareness.

Not to spoil anything but a large part of the plot was how it circumvented that using a method not entirely unlike this. Life imitates art imitates life I guess.

I agree about the part where it's a static model - it's effectively the same as using a notebook without memorising what's in the notebook. That doesnt mean it can't generate insightful commentary as a result of a novel way of interacting with itself though, I think?

But with secondary and tertiary modifiable and searchable memory systems like MemGPT, the sum of the parts is not static, and can even introspect and reflect on and suggest modifications to its own code and protocols.

It's already learned so much that in many cases it's just a matter of addressing and combining what it already knows, with what new knowledge and instructions you can fit in its token window.

See my post about my experience with MemGPT debugging itself and suggesting improvements:

https://news.ycombinator.com/item?id=38344033

This approach to recursive feedback and self inquiry, and other comments in this thread, vividly remind me of my friend Jim Crutchfield's work on feedback and chaos theory, and his amazing video with groovy music, "Space-Time Dynamics in Video Feedback":

https://www.youtube.com/watch?v=B4Kn3djJMCE

A film by Jim Crutchfield, Entropy Productions, Santa Cruz (1984). Original U-matic video transferred to digital video. 16 minutes.

See https://csc.ucdavis.edu/~chaos/chaos/films.htm

Citation: J. P. Crutchfield, "Space-Time Dynamics in Video Feedback". Physica 10D (1984) 229-245.

https://csc.ucdavis.edu/~chaos/chaos/pubs/stdvf-title.html

Possible spatio-temporal attractors:

1. Fixed point or equilibrium

2. Limit cycle or periodic oscillation

3. Chaotic attractor: (spatially coherent or incoherent)

4. Noise-driven spatial structures

The self awareness (such as it is) is in the prompt (and the changes in the prompt from one iteration to the next), not the model.

Of course, you could use these self-chats back into the training of the model. Maybe the result would be a little more open-ended.

As far as I understand it, that would result in a very weak and local manifestation of it when baked into a model, because of its size. But as usual with this tech, one has to try and do before making claims.

> Learning requires a model that grows. So there can't be any learning, by definition.

Do you also believe human brains need to "grow" to learn? Whatever you mean by "grow", it doesn't seem like a rigid definition.

Awareness means being able to say "I don't know." And none of these models will ever eject that answer.

The reason: The training set does not include "I don't know." Everything on the net is confidently right or, more often, confidently wrong.

LLM is Artificial "Bullshit" rather than Artificial "Intelligence".

Every time we make advances in AI, what we discover is that humans have less intelligence than we ascribe rather than machines have more.

> Every time we make advances in AI, what we discover is that humans have less intelligence than we ascribe rather than machines have more.

I don't think your cynicism is supportable as stated. Humans are as intelligent as they are, and LLM's are not bullshit. They are doing more and more flexible and useful things that were not automatable before, or even predicted to be automated any time soon.

However, I do agree that human thinking is progressively being demonstrated to be amenable to relatively simple algorithms applied to lots of data and parameters.

That was the only likely way things were going to play out. Sooner or later the slow process of demystification started by human's first attempts at introspection was going to give way to something as automated ... as our own neural processes.

> things that were not automatable before

Where the automated result now has error bars on it though.

Typically when we automated things in the past, that meant getting extremely high success rates. We're accepting lower success rates to automate things that were previously not automatable, by using LLMs. Thats not quite the same thing, even if it does have value!

If there's anything we can be sure of, it's that LLMs are bullshit generators that happen to be correct a certain percent of the time. Not just bullshit generators. Plausibly sounding bullshit generators.

This is totally wrong, GPT-4 knows what it knows and doesn't know to a good degree of precision. Look at the calibration graphs on this page: https://openai.com/research/gpt-4