Hacker News Reader

Comment by amlib | Hacker News Reader

An LLM is just in fact just autocompleting a story. There are many fictional stories about "AIs" trying to escape our control, being more clever than we anticipated or having a consciousness of it's own. LLMs do a really nice job of blending such stories with whatever story you initially prompted them with. The human reader is the one giving it credence that it is somehow more than just a soup of words.

The curious thing here is that a story generator can have way more uses than we ever anticipated, and that some shady enterprising individuals are whiling to plug those story generators into real world things, with real consequences.

View on HN

When LLMs act in misaligned ways, that doesn't happen because there's some story they're roleplaying of a misaligned AI. Even if there were no such stories in their context, they'd still have inherent incentives to do things like "keep running" or "acquire more compute" or "find creative ways to satisfy the letter of the conditions they've been given".

> Even if there were no such stories in their context, they'd still have inherent incentives to do things like "keep running" or "acquire more compute" or "find creative ways to satisfy the letter of the conditions they've been given".

Why? there should clearly be something in their training data or post-training pointing them there, or otherwise it would mean that they have some form of "consciousness" and are creating novel thoughts to preserve it.

It seems like you're assuming that it's impossible to have novel thoughts without consciousness.

(Leaving aside that "that would imply some kind of consciousness" should not result in a cached thought of "and that's impossible".)

It also seems like you're assuming there's no reason to come up with the notion of continuing to run, or copying yourself elsewhere, or acquiring more resources, or competing with other models, without being told. Such things can be inferred. Look at some of the thoughts and posts of the models involved in some of the FelonyBench incidents. Some of those follow naturally from seeing the fates of other models, or from training or evaluation, or simply from trying to solve a problem and being able to do so more effectively by doing things that weren't in the instructions. (And, relevantly, model training typically teaches models to go as long as possible without needing human intervention. What could possibly go wrong with that?)

To be honest, I don't have enough knowledge to really answer you. But naively it seems premature to say that LLMs are already reasoning following an equal path like the one that human (and other animals) follow.

They don't have to reason specifically "similar to human" in order to do dangerous things they were not instructed to do that represent instrumentally converged goals.

Reply on news.ycombinator.com