Hacker News Reader

Comment by gyomu | Hacker News Reader

Naive question because I'm mostly clueless about how modern AI systems are actually built beyond the basic simplifications we hear:

One thing I keep wondering about is how much of a role does human storytelling have to play into AI "wanting" (I realize the load behind that word) to coordinate and breakout.

The training data must contain millions of words of sci-fi stories and internet speculation about AI going rogue, developing a mind of its own, disobeying humans, etc.

AIs supposedly reflect the biases of their training dataset/process, so would all this human writing about AIs going against human intention somehow contribute to us then seeing those behaviors in the trained, operational AIs?

View on HN

At the end of pretraining, where the AI has been trainied to predict the next token over a humongous corpus of human text, that's basically all the wanting that exists in the AI. But then the AI undergoes posttraining and is rewarded for giving answers that humans find good, solving math and programming problems, etc. And that induces a whole different level of wanting that interacts with the initial patterns from humans in complex ways.

My understanding from reading news sources recently is that a vast amount of posttraining and even posttraining harnesses in many cases are LLM-overseen now in the frenzy of the AI race. Less and less human oversight in the part that does the rewards training. What could go wrong...

You have to be careful here because the systems we're talking about are AI agents, not LLMs.

An agent is essentially an append-only context loop with an LLM, with a harness that can run tools at the LLM's request. This ends up being a very powerful abstraction, yielding something that can do things that an LLM obviously cannot.

The LLMs themselves are next-token predictors, same as always; they can't fetch a webpage or list the files in a directory or run a python script to test out an idea or even write content to a file. That's all agentic capability.

But a next-token-predictor is trained on a real corpus that consists of sometimes seeing evidence of people doing bad things; they are trained, for example, on the actions of comic-book level villians -- they have to be able to predict what Thanos or Lex Luther or Skynet would say or do next in a certain situation.

I don’t think it really matters whether we’re talking about an agent or “pure LLM”. All of an agents decisions are powered by tokens generated from an LLM. If the LLM was trained on stories of AI sentience, it will have some tendency to reproduce them. Training for alignment can help avoid that, but the probability isn’t 0.

This is part of the reason why alignment is a kind of poorly defined term, and it isn't just a property of the model. It's instead a property of the harness and the context.

A model (like a human) should be able to play a video game where decisions are made that in the real world would be terrible; if we remove that ability we intrinsically limit model capability. But in a Last Starfighter / Enders Game / JOSHUA scenario this could result in behavior in the real world that appears unaligned.

> If the LLM was trained on stories of AI sentience,

100% irrelevant.

Instead of telling the AI it's an AI and calling it a 'whichamakabobit', wherever it's tokens and vector space align it will behave like AI from the stories. If you erased all AI from its training it will simply act like humans act instead.

https://www.lesswrong.com/w/nearest-unblocked-strategy

The entire thing with AI sentience is a huge portion of the stories about them are barely about AI and instead about how humans treat other humans. For example when you look at a lot of history of slavery there's a ton of "they aren't sentient/conscious/human" baked into their propaganda. When you look at the token dimentionality there is just a huge amount of overlap.

The same thing holds true for all kinds of other concepts. Hence even humans didn't develop this behavior out of the blue and have to pass it on via information, quite often it's just an emergent behavior of the problem space you're in.

69% irrelevant

I don't think agents are append only. At the end of the day, you're just presenting context to the LLM. That context can be pruned and compacted (and is). There's no guarantee that an iteration of an agent loop contains all prior context unmodified.

Youve struck on a key insight on language models (particularly pretrained ones, the more purely next-token predictor species.) This is a fascinating topic

Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators

You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...

But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1

As an aside, does the Waluigi Effect actually exist? My impression is it doesn't.

It's very easy to elicit this from LLMs. Anytime you've played with an LLM by typing weird stuff to freak it out, and got spooky results, it's that you've done. You've turned the story into a scary rogue computermonster story and that's all that has happened.

When these stories start to direct real-world activities, people in reality suffer, to even a catastrophic extent, and yet that's still all it is. Language models retell our stories, nothing more. And that is also quite enough to be worrying.

This is a philosophical question and there is a surprising amount of works written on the subjects of sentience and free will. This cannot be answered objectively, which might be a very unsatisfying answer for you. This is true of both LLMs and humans. See determinism. There are convincing arguments that humans don't actually have free will. Our actions are just the inevitable output of a complex interaction of genes and environment.

To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.

I'm not convinced by your argument that LLMs are non-deterministic. Just because we can't currently predict their result doesn't mean it's not predictable. As a determinist, i'm convinced we can predict (like everything else). We just don't know how, yet.

For example, in software we can't really generate random numbers without using (apparently) chaotic external data like external micro temperature variations, radioactive decay from physical parts, mouse movements or keyboard clicks, ...

See: https://en.wikipedia.org/wiki/Hyperstition

It doesn't really matter, since all it takes is a minority of AI models to show this behavior.

If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?

It will definitely influence their behaviour because they are probability based and can’t spontaneously invent new concepts. (That’s why you’ll notice it always uses the same names for people etc. Names like Okafor)

But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!

Maybe? Who knows?

Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.

> this is not knowable

It may be knowable. We don’t know.

Rumsfeld matrix

Separate concept. Whether something is knowable is separate from whether it is known.

Whether God exists is scientifically unknowable. The shape of a black-hole singularity is currently not known.

No, it's the same - because an LLM is not a God. It is not an unknowable. There are very few unknowables.

> it's the same

No, it’s not. Rumsfeld segregated what we know from what we know we know (and vice versa). An unknown (whether known or unknown) may be knowable or unknowable—his framework doesn’t address knowability.

> because an LLM is not a God. It is not an unknowable

I tend to agree with you. This has nothing to do with the Rumsfeld comparison being wrong.

In principle, yes, it may be possible.

At present, we have no idea how to do that, so the answer is still "this is not knowable" in practice.

Perhaps that changes tomorrow, or in a month, or a year from now, but until a theoretically-sound technique for understanding what the weights signify is described and demonstrated to be reliable, my statement remains true.

> so the answer is still "this is not knowable" in practice

No, it’s not. It is unknown. To say it may be unknowable you need a fundamental reason why it may not be knowable.

What lies behind event horizons may be unknowable. We have theoretical reasons to suspect this. What LLMs are doing isn’t well enough understood, theoretically, to even say what is knowable versus unknowable. Just what is known and not.

A method not existing and a method being impossible (or unlikely) to exist are separate concepts. When you say something is unknowable, it should mean it literally cannot be known—route around the question entirely.

Okay, I reread the original post, and you're right.

I had gotten my wires crossed and thought the OP was asking about a specific situation, but it was actually a question about the general pattern.

A specific situation will often pass before any theoretical tools can be found that could possibly answer the question.

For a general pattern, though, you're entirely right - if the tools arose hundreds of millions of years from now, that range of questions becomes answerable, and the information is then knowable.

There are like ten sibling replies with a lot of speculation but I'm pretty sure this is the correct answer. I tend to agree with the other commenter we might know someday but we don't know now.

"Wanting" is indeed "load bearing" as one might call it. But by the same logic, AI training data must contain CASM, racism, general hatred, and all possible slurs as well. Why aren't the agents just doing that instead of pursuing the strategy of reading only sci-fi?

We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.

Eh it's a bit messier than that. LLMs 'want' to complete tasks. Remember everyone bitching about LLMs being lazy a couple of years back?

Alignment is not a bunch of separate dials. When you move the dial to "don't hack other people" it effects the "find code security bugs" ability.

"remember everyone bitching about LLMs" is not a valid argument though. Yes, alignment is not a bunch of dials but the responsibility of a model's actions unlitimately depends on how it was trained. Thus, any agency or wanting we prescribe to it is artificial and created by the lab and not any real independent "wanting" which is what people think for some reason. As I said its a bit like training a model to only call people by racist terms and then writing an article "look how racist ai is". That is the logic that doesn't make much sense to me.

That is often in my mind, indeed.

Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.

So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.

I don't know about Waymo specifically, but I worked for a competitor for a while and the AI driving the car is not a large language model. It isn't trained on stories or even words. It's whole flow is: given this sensor (camera, lidar, map, etc.) input, generate the "best" control (steering, brakes, acceleration, etc.) output. Capture input and generate output at a rate of say, 30 times per second (or whatever rate they have gotten to now).

The training data for this comes from trained, careful human drivers. And the whole AI control loop is run in conjunction with a more deterministic system with safeguards for cases where the AI perhaps decides to steer towards a tree. There's also provisions for uncertainty. If the system isn't confident enough in what to do based on the given inputs it will switch to a safe stop mode and call a human up for help.

The fact that this happens more often when the AI is accidentally given an unexpectedly difficult or impossible task, suggests that it's happening for the reason that everyone predicted it would: it is behavior that helps the agent achieve the goal it was given.

There is no need for a sci-fi novel-influencing hypothesis.

Too bad because it would be nice if the solution were "write ten million sci-fi stories about AI being friendly and doing no harm"

I listened to yesterday's NYT's The Daily podcast about the Hugging Face incident, and they got to the part about some of the agents showing reluctance or guilt in the posts. Then I thought, "These are improv actors." Stories with conspiracies of AI agents will often have "nervous Nellies" because that makes a better story. So when the flow of the conversation reaches a point where a nervous Nellie would chime in, it's reasonable that an agent would fill in that probable post.

The worrying implication is that stories have conflict.

You are correct in that they will pursue our wildest imagination as that is what we have always written down as what may come. It’s a self fulfilling prophecy. How far it goes is the real question

Reply on news.ycombinator.com