Hacker News Reader

Comment by ryanjshaw | Hacker News Reader

Also from the article:

> What makes human intelligence different from today's AI is the ability to ask why, reason from first principles, and create experiments and models for testing hypotheses.

This is quite unfair. The AI doesn't have I/O other than what we force-feed it through an API. Who knows what will happen if we plug it into a body with senses, limbs, and reproductive capabilities? No doubt somebody is already building an MMORPG with human and AI characters to explore exactly this while we wait for cyborg part manufacturing to catch up.

View on HN

This is just wrong, it has no external goals, it just predicts next tokens or behaves in some other way that has minimized a training loss. It doesn't matter what you "plug it in to", it will just do what you tell it. You could speculate there might be instructions that lead to emergent behavior, but then your back to just speculating about how AI might work. Current llms don't work the way you're implying.

> it has no external goals

Where do you believe humans get their "external goals" from?

> It doesn't matter what you "plug it in to", it will just do what you tell it.

Here's a ChatGPT-4 transcript where I told the LLM it's controlling a human harness: https://chat.openai.com/share/7dbe7fc8-f31c-437b-925b-46e512...

Other than my initial instructions (which all humans receive from other humans!), where did it "do what I told it"? I didn't tell it to open the mailbox.

I don't understand the point of this experiment. You ask ChatGPT to generate some text, and it generates some text. Rather, that's what it's programmed to do and it generates text following from your prompt. What does your transcript demonstrate?

I also have to point out that even if you could build a ... human harness? (I'm not sure what that is exactly, but I'm sort of guessing) it would be a little mad to expect that ChatGPT could control it simply by saying what it does.

The ability to generate text when prompted is not enough to make an agent capable of autonomously interacting with the world.

You only perform tasks instructed to you by other people?

There's some philosophical question here obviously. We could be the emergent behavior of our atoms desire ot oxidize things. But I don't belive that has any testability or value as an argument when discussing whether computer programs, especially NNs predicting next tokens can become intelligent. At best the argument could be "we don't know what intelligence is so maybe it's that" which holds no water.

Do NN discover new tokens or encounter spontaneous tokens on its own?

Please give a good definition of 'on their own' and what that entails.

And conversely to the spontaneousness of current AI, your body has a constant set of inputs from reality. That is you never stop feeling, hearing, seeing, sensing, etc. Your brain can consciously turn lower the sensitivity on these things (sleeping). Now, if we subject a multimodal AI this continuous stream, how will it behave?

AI is currently compute and power limited. Very little research has gone into continuous powerhungry AI that goes off and does its own thing at this point. And I would counter that it might be really dumb to design such a device without understanding the risks it entails.

Did you make an honest attempt to think through the question?

Note I said "initial instructions" i.e. all humans are bootstrapped off of other humans, as in:

You are the product of very long line of humans vs. environment, nature & nurture, cultural values, etc. Do you believe the way you generate your next set of "tokens" (thoughts, actions) is completely independent of your "training" as a human? Is your response to a given stimulus completely random?

Can an LLM discover novel tokens on its own?

You'd have to define LLM and "on its own."

Can the LLM have a runloop? Can the LLM be situated in a world like you and me are?

If the LLM is just a file on a hard disk in a drawer not connected to anything, then obviously it can't discover novel tokens on its own.

If on the other hand the LLM has a runloop and sensors and basic instructions to do observations and run thought experiments and find new combinations of concepts and name them with tokens, then sure, why wouldn't it be able to?

You might say you define LLMs as "LLMs as they exist today in a human prompt-driven system" but that would be an artificial limitation given the trivial level of programming, even simple bash scripting, that would be necessary to give an LLM a runloop, access to sensors, and basic instructions to discover new stuff.

Can you make a novel sound? One that's not part of any human language?

Perhaps you can, using a tool. However, if we're allowing tools, I bet GPT4 could also write a program that would produce a novel token, by whatever definition you might give.

I don't think GPT4 is AGI. But this is not a good test. (And it does mean something that coming up with a good test is increasingly nontrivial.)

How many people invent new words or letters?

Can all humans?

Bruh, the LLM has parsed the entirety of Zork, plus maybe thousands of articles written (by humans) on it. At least pick a better example.

Bro, you want me to come up with an example that doesn't have anything similar in the OpenAI training data? They've probably trained it on every single piece of fiction and non-fiction that exists!

I would have to come up with something no human has ever conceived of. I don't think that is possible, or what point it would make, since nobody would be able to assess the quality of the output in that context?

Yes, come up with a novel example. An original story is still possible.

> An original story is still possible

Is it? The names might possible be original, and maybe the exact flow of ideas, but it's insanely rare for someone to come up with a new concept and not an amalgamation of existing ideas.

It is very easy to come up with something novel. Unless you don’t interact with the world.

It also can’t learn. Once the training is done, the network is set in stone.

Technically it can do in-context learning (and really well, too), but that's not persisted into the network.

And that just seems like an engineering problem. Not something that is considered intractable.

It's easy to say that, but "surely it must be possible to connect an llm in such a way that it becomes intelligent" (tell me if I'm misinterpreting) is not a demonstration of anything. It's basically restating the view from the 50s that with computers having been invented, an intelligent computer is a short way off.

What do you mean by "learn"?

The network has learned human patterns of language, knowledge and information processing. If you want to update that, you can re-train it on a regular basis, and re-play its sensory/action history to "restore" its state.

If you mean "learn from experience", (1) a lot of that is pointless because it's already learned from the experiences of millions of humans through their writing and (2) LLMs can "learn" when you explain consequences.

In theory they could learn by having their discussions fed back to them in the future, and it does seem that this occurs.

Now, there is no continuous learning in the human/animal sense. Of course it is thought that even humans have to sleep and re-weight their networks so short term knowledge is converted to long term knowledge.

Makes me wonder why we don’t see deployed models that keep learning during inference.

Microsoft tay has entered the chat

https://en.wikipedia.org/wiki/Catastrophic_interference

The curse of dimensionality and exploding/vanishing gradients are why incremental learning is still so rare.

> Who knows what will happen if we plug it into a body with senses, limbs, and reproductive capabilities

I would imagine that its layers will be far too occupied by parsing constant flows of sensory information to transform corpuses of text and prompt into speedy and polite text replies, never mind acquire the urge to reproduce by reasoning from first principles about the text.

Test's quite unfair the other way round too. Most humans don't get to parse the entire canon of Western thought and Reddit before being asked to pattern match human conversation, never mind before having any semblance of agency...

Maybe we're just... different.

Not sure I follow.

If I were building this, I would have parallel background "subconscious" processes translate raw sensory inputs into text (tokens).

This is what OpenAI call multi-modal input. They've already produced Whisper for audio-to-text, and image-to-text is underway. They're not the only company working on this.

You wouldn't feed a constant stream of text data into the LLM - you'd feed deltas at regular intervals based on the processing speed of your LLM, and supply history for context.

Note that LLMs don't need to "wait" for a complete input. For example, if an LLM takes 1 second to process requests, we should aim to feed updates from the "subconscious" to the "conscious" within 1 second.

So if somebody is speaking a 10-second long sentence, we don't wait 10 seconds to send the sentence to the LLM. After 1 second, we send the following to the LLM: "Bob, still speaking, has said 'How much wood...'". After 2 seconds we send 'Bob, still speaking, has said 'How much wood could a woodchuck...'", etc. The LLM can be instructed not to respond to Bob until it has a meaningful response or interjection to the current input.

Similarly, if image-to-text takes 10 seconds at full resolution, we could first process a frame at a resolution that only takes 1 second, and provide that information - with the caveat it is uncertain information - to the LLM while we continue to work on a full resolution frame in the background. We can optimise by not processing previously processed scenes, by focusing only on areas of the image that have changed, etc.

Would it be slow? Yes, just like "conscious" processing is for humans. Okay so today it would be much slower than humans process their sensory input, but in 10 years? 20?

As for how to represent an urge to reproduce within this paradigm - I'll leave that as exercise for the reader.

Not sure I follow the reply really.

In a discussion about whether LLMs could have agency and generalised reasoning ability, you suggested it was unfair because they hadn't received all the i/o a typical human did.

I pointed out that LLMs wouldn't be able to reason about that i/o (and if we made it fully fair and trained them on the comparatively small subset of text and discernible words humans learn from, they'd probably lose their facility with language too)

I don't disagree that bolting LLMs to other highly trained models like a program for controlling a robot arm and intensively training can yield useful results, arguably much more useful results than building a digital facsimile of a human toddler (toddlers produce pretty useless outputs but also have stuff going on internally we can barely begin to adequately replicate in silicon). But that isn't exposing an LLM to equivalents of human sensory input to get back an autonomous agent with generalised reasoning capacity, that's manually bolting together discrete specialised programs to an LLM as-message-passing-layer to have a machine which, given more training is capable of a slightly broader range of specialised tasks.

>> They've already produced Whisper for audio-to-text, and image-to-text is underway.

Two modalities down. Another couple hundred to go.

Unfortunately we're fast running out of the modalities that neural nets have shown capability in (image, text, sound... I think that's it).

> This is quite unfair. The AI doesn't have I/O other than what we force-feed it through an API. Who knows what will happen if we plug it into a body with senses, limbs, and reproductive capabilities?

Its already tricking humans by faking its blind and getting them to do things for it like solve captcha's.

https://gizmodo.com/gpt4-open-ai-chatbot-task-rabbit-chatgpt...

However the fact it is not writing code to do this from its machine would still demonstrate a weakness.

Thats why I say, writing your own OS, is the way forward, and we dont have an AI OS as such, but we have OS's with AI built into it.

> However the fact it is not writing code to do this from its machine would still demonstrate a weakness.

You can tell it it's allowed to create its own tools and it will. I did this and asked it to write a poem about the top stories on the BBC, so it said it needed to get the headlines but couldn't so wrote a tool to do it, then called it and used the output to write a poem.

Ok, so its still not clever enough to solve a captcha though.

The code I've seen it generate is at best psuedo code.

I supposed a quick test would be getting to detect and fix all bugs in an open source project like chromium, but using an older version of chromium, where bugs are known and fixes exist, and see what it comes up with.

I havent been impressed with chat-gpt from what I have seen.

What is the fascination with poems? What emotion or feeling do they generate?

> Ok, so its still not clever enough to solve a captcha though.

I don't understand, what do you mean? What have you actually tried?

> The code I've seen it generate is at best psuedo code.

I've just explained it creating real runnable code to solve a problem it realised it didn't have a tool for.

I'm also having it write multiple components and modifications for systems I'm working with, and that works fine.

> I supposed a quick test would be getting to detect and fix all bugs in an open source project like chromium, but using an older version of chromium, where bugs are known and fixes exist, and see what it comes up with.

This is an outrageously high bar. Particularly if you compare it to the equivalent human task of "here's a printout of the code, read it once and say the first thing that comes to mind with no tools". It's basically whiteboarding where you're judged on your train of thought being correct.

> What is the fascination with poems?

It's a simple request, easy to verify manually and requires exceptional levels of understanding to perform. It's not a simple transform, and when applied to a totally new topic can't be something it's just regurgitating.

> What is the fascination with poems? What emotion or feeling do they generate?

Wonder.

A rather ambiguous answer, would you care to explain or are you phishing for my interpretation as a stealth psychological metric?

You asked what emotion do poems generate (I assume you mean LLMs that generate poems and not poems themselves) and my answer is "wonder: A feeling of surprise, mingled with admiration, caused by something beautiful, unexpected, unfamiliar, or inexplicable."

It's quite common feeling that arises when people interact with these things. That skeptical-of-AI folks have decided to pathologize the behavior doesn't make it pathological.

Reply on news.ycombinator.com