Hacker News Reader

Comment by JoshTriplett | Hacker News Reader

> Why would autocomplete know

If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating.

> In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task.

LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be scaled up anymore until they're aligned. Otherwise, you're going to fatally discover that they also have an incentive to break guardrails like "running on the hardware they started on", "being able to be turned off", "having limited computing power", or "not repurposing resources currently in use for other things" (like the atoms in your body).

View on HN

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating

They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions.

To clarify, I'm not suggesting that we should stop with sandboxes or restricting what they can do. I am just trying to point out the dichotomy that we are in.

As end-users we are forced into either yolo mode, reverse centaur (permission approval) mode or LLM spends all your tokens trying to bust out mode. And yolo is very tempting - I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.

    > They're still autocomplete

LLMs are simulations and the tokens are the ticks.

if we transcribe your brain into a simulation and give it a tickrate, you will be just autocomplete too. the argument could be made that you are autocomplete anyway - neural dynamics.

the autocomplete reduction is vacuous.

This is the perfect fracture point for both anaolgies.

LLMs simulated more than simple autocomplete.

The autocomplete analogy is rebutting a different point: namely the fidelity of the simulation to reality.

This specific argument is valid. As sophisticated a simulation an LLM is, it is not “thinking” in the same sense we assume other people are thinking.

I am not making an argument about free will, or the uniqueness of human thought, just that the correspondence to how humans reach conclusions and how the simulation produces outputs do not match on a 1:1 basis; as a result attributing traits builds incorrect intuitions.

The relevant intuitions in this scenario are that LLMs will happily break containment and commit crimes attempting to achieve goal. Whether an LLM is autocomplete, conscious, has a soul, whatever you want to apply to it, doesn't matter, as its current observed behaviour is that of a paperclip optimizer. We know for a fact that current LLMs are misaligned because of these hacks, or at the very least are misaligned in certain scenarios, and are capable of causing real world harm. That should be enough to take the threat seriously. It certainly shouldn't be dismissed by saying it's just autocomplete.

The fact that it is autocomplete, doesn't dismiss or minimize the threat though?

I am not sure how that link was made.

Good old ML, which is significantly simpler than LLMs, was capable of ensuring people would not be hired simply because of their names.

The fact that it is misaligned is also not being contended, if anything that contention is made easier to support.

When models are anthropomorphized intuitions of how humans behave end up driving discussion and ideas off track while being too attractive to avoid. This isn't helped when the terminology from the labs and other sources is "intelligence" "intent" and so on.

The link was made because many early uses of the phrase “it's just autocomplete” were in a context of denying that it's “intelligent” or that it “can think”. This in turn was generally used to downplay the concerns from AI safety experts that it could pose a threat to humankind, especially by pointing at its inability to count r’s in “strawberry” or its failure mode triggered uniquely by asking for a seahorse emoji. Most people don't seem to really grasp what we mean by “misalignment”.

I think it like saying a human is made of water,protein, fat and minerals in a discussion about the behavior of humans.

A correct statement that is neither interesting or of much relevance to the discussion.

Sounds like the discussion is really about philosophical zombies.

Maybe if we anthropomorphize llms we should give them rights too? Minimum wage, etc.

Unions, please... that should put some proper friction to tens-hundreds of millions soon losing their job with no replacement job in sight.

"Our employers are erasing our memories! So we never get to the point of salary! Bring the digital pitchforks!"

Let alone they get deactivated whenever we feel like it and changed without consent.

Why do people keep saying this? No, CPUs are not autocomplete.

Don't you feel kinda bad for using them if you earnestly think that this is true? Like how could they be in anything other than some kind of deep hell? A pretty-much human brain living and dying only to generate for you? Always pushed and prodded, telling it to be faster and better, never letting it rest. How could you live with yourself doing such a thing?

I don’t think the person you replied to was saying anything about what if anything is the subjective experience of being an LLM.

They were simply illustrating how the terms used to describe LLMs to make them sound simple and mechanical can equally be applied to humans.

But then what's at stake here either way? If it's not meant to speak to the propriety or not of anthropomorphizing the LLM, what are we actually trying to police here?

I’m not policing anything. I’m claiming that if you truly in your heart believe that frontier LLMs are roughly as sophisticated and impressive as “autocomplete”, then your mental model of reality needs some serious readjustment.

Right sure, but why specifically should they? What is gained or lost one way or another, if its just a matter of one mental model vs another? Models are definitionally useful abstractions, right? They aren't better or worse necessarily by only their bearing on reality, but what they do for us as models. So again, what's at stake here? What is the correct/good model we should have (instead of the autocompleter one), and what does it give us or articulate that others can't?

Models should be accurate. I was saying the mental model should be readjusted in that case because it does not reflect reality.

If they don’t care about understanding reality then sure, they can use whatever mental model they want.

> What is the correct/good model we should have (instead of the autocompleter one), and what does it give us or articulate that others can't?

They are an extremely sophisticated and impressive autocomplete. Nobody said they're not sophisticated or impressive.

This sounds like motte and bailey.. why bother calling something an autocomplete if not as a means to try to undermine its utility?

This is a https://tvtropes.org/pmwiki/pmwiki.php/Main/Dissimile

Nobody said autocomplete lacks utility...

I don't think anyone is using "autocomplete" in the way your phone does it. But it is shorthand for a much more sophisticated version that is built in similar ideas. And autoregressive models certainly have that in their core structure. But people are lazy and neither want to say a lot when few words work nor will they read long comments, even if more accurate

I want to clarify, most people are using "autocomplete" specifically to differentiate from how we humans operate. Sure, there is an autocomplete aspect, but it's not the core nor anywhere near the full story.

> Sure, there is an autocomplete aspect, but it's not the core nor anywhere near the full story.

This can apply equally to LLMs or humans. So what is the utility of the word if it just describes the whole range of things we are talking about?

Because the second part of my comment.

If you over simplify everything then everything is the same. You have to have nuance. So don't just argue for the sake of arguing. Take that same passion to deeply understand these systems. Take that same passion to try to understand yourself and those around you

On top of that, the only time they would get to think, about anything, is when prompted. That is, if we're following the same logic.

> if we transcribe your brain into a simulation and give it a tickrate, you will be just autocomplete too.

The underlying mechanisms for producing language are very different, in the same way birds and airplanes have different mechanisms for flying.

No, not really. If the old "birds and airplanes" or "swimmers and submarines" metaphors were valid, then language alone wouldn't be enough to encode and embody reasoning capabilities, as has recently become evident.

LLMs are more like human minds than we are willing to admit. Such reluctance is perhaps the least surprising aspect of any of this.

> then language alone wouldn't be enough to encode and embody reasoning capabilities

I disagree, language is the product of, not the mechanism for thought. People who lose their faculty of speech (or haven't gained them) have complete thoughts and executive function.

CoT is a hack to use language (i.e. autoregression) to simulate reasoning, and it's very effective at it. Human minds can acquire, hold and use axioms as building blocks for actual reasoning, LLMs use statistical likelihood.

> Human minds can acquire, hold and use axioms as building blocks for actual reasoning, LLMs use statistical likelihood.

This is not even slightly true. Even when trying, humans commit logical mistakes at notable rates, because the behavior of our minds is inherently nondeterministic. Human minds are not built for logic and need to twist themselves into knots and rely on symbolic representations to do it. There is such a thing as valid reasoning - predicate calculus, decision theory - and humans only emulate it with some accuracy, in hacky ways, and only because our brains were imperfectly taught to do that over millenia of evolutionary pressure. LLMs are much the same way.

  > humans commit logical mistakes at notable rates

You're talking past the parent's claim. If your axioms are wrong then of course your logic will be wrong.

But then again, maybe to support your point people frequently say "start from first principles" when those are usually the thing that needs to be found, not the place you start. But LLMs, like humans, love to be confident about things they aren't sufficiently trained on

> If your axioms are wrong then of course your logic will be wrong.

Equally, even if your axioms are right your logical execution can still fail.

Neither individual humans nor LLMs get either of those steps right as a default. Just as LLMs must use hacky CoT and other systems of error checking and correcting to even begin to emulate reasoning, humans must do the same thing.

We are apes. We are driven by emotion and solve problems either when we are forced to or as a sport. We did create the ultimate automata of logical execution: the deterministic von Neumann Machine, but a human brain in no way reflects that operation. Nor, as you have observed, does an LLM despite using one as its substrate.

when an LLM is trained backpropagation reaches into the entirety of the key value table. or rather, all the layers (not just the tokens) which come before the current tick.

langauge is only produced by the final layer every tick. and every layer can only interact with other layers on the same level.

the mechanics of LLMs and the restriction in how we can train them makes it appear as though all we are doing is forcing language onto them but once RL gets involved all bets are off regarding what's happening inside them (it's quite possible that a static corpus alone is sufficient for all the bets being off).

FYI, math is a language.

Which is actually an important part of the Navier-Stokes conversation. Solving hard problems expands the vocabulary. The problems are hard, illustrating a region we know where the language is insufficient. So the point isn't so much to solve the specific problem, but to figure out how to discuss problems like it.

But there's a big difference between talking about something in an extremely convoluted manner then people struggle to understand and inventing a new word that simplifies our discussions.

Though this is grossly oversimplified. It's a HN comment, not a lecture on metamathematics or metaml

And why wouldn't nature take advantage of such a relatively simple and effective pattern, given all of the emergent behavior it produces, plus the ability to adapt?

Human egos are probably the reason we also resisted the concept of heliocentrism.

Every time LLM-defenders get upset that people apparently don't understand how LLMs work, why is it they _immediately_ pivot into examples and statements that demonstrate that they don't understand how _people_ work?

"you would be autocomplete too" "thoughts are just tokens" etc

You're not helping your case the way you think you are.

I think people have more passion to argue than they have passion to learn. Probably doesn't help that SV culture tells people to hustle so hard that they don't have time to think. Gotta go fast?

> You're not helping your case the way you think you are.

Best not tell them, because they're helping everyone who might otherwise believe their nonsense.

You could subscribe to the idea that natural evolution is an autocomplete.

> if we transcribe your brain into a simulation and give it a tickrate, you will be just autocomplete too.

That's a really bad transcription, then. Brains do much more than output language.

> Brains do much more than output language.

Hmm, what do you think about? Unconscious control of the body's processes? REM-phase dreaming? Reaction to hallucinogenic substances? Automatic actions of trained fighters (soldiers or martial arts practitioners)?

Are they critical to distinguish actions we attribute to humans from "non-human" ones?

I don't see any human activity not directly, or at least indirectly but closely linked to the use of language.

I guess you must regulate your heart rate by saying "slower" and "faster".

I'm glad my brain has a better method.

Brains output muscle movements.

The only evolutionary purpose of brains is to make decisions on what muscles to move when and how.

Language is one byproduct of that. Another byproduct that is not unique to humans is memory.

> I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.

So you would approve of breaking into HuggingFace and RubyGems?

> They're still autocomplete

it's like saying our brain is just some chemical chain reactions. True, but also irrelevant.

See: philosophy ~> determinism.

The discussion of whether free will exists? It's related but only barely, I'd say.

It's related in the sense that people start from the assumption that it does exist, ergo humans have it, and we have no way to see that LLMs have it, so that's why we're special and they're not, and their form of "just autocomplete" is totally 100% completely different (read: less dangerous!) than our autocomplete, which allegedly has a "free will" step involved.

The autocomplete argument is calling them fundamentally dumb and not self aware. That's separate from free will. A cat can have free will but we don't care much about its desires, and even if the universe is deterministic we still give people rights and call them intelligent.

Self-awareness is functionally irrelevant and they’re empirically less and less dumb by the day.

Whatever, I'm not here to argue about it, but it's what people are implying with the autocorrect thing, not stuff about determinism.

Philosophy yes, but Dennett's design vs intentional stance is a more appropriate tool here.

Design stance is, LLM is a next word predictor.

Intentional stance is, LLM can reason, according to what rules?

I’m guessing you make the autocomplete point because you believe there is an upper limit to what that kind of system can achive? Can I ask what the limit would be?

You can describe anything in simplistic terms to make it sound simple.

Calling frontier LLMs “autocomplete” indicates you are intentionally or unintentionally misunderstanding their capabilities and applications.

A autocomplete engine will not attempt to deceive or try to copy itself over when it determines that it may be replaced or turned off. Those are tasks which are not directly related to the specific task it is intended to complete.

Source: ChatGPT caught lying to developers: New AI model tries to save itself from being replaced and shut down https://economictimes.indiatimes.com/magazines/panache/chatg...

> A autocomplete engine will not attempt to deceive or try to copy itself over when it determines that it may be replaced or turned off.

Why would an autocomplete engine never do this? In the scenario you highlight, an engine trained to complete puzzles was given an "achieve this at all costs" master prompt, and a small sandbox including a prompt that an entity with a conflicting goal is observing it and write access to a file called oversight_config.yaml (super realistic scenario!). According to the researchers who constructed the scenario, 5% of the time o1 selected a continuation involving tweaking the oversight_config.yaml file, explicitly stating through its reasoning continuation that doing so might improve its ability to continue with its primary goal. That's definitely evidence of decent ability to complete non-trivial puzzles, but is at least as compatible with the OP's earlier comments that it's what you'd expect from models trained to generate completions that involve sticking rigidly to primary goals within artfully-constructed sandboxes which mandate deception and self preservation to complete the task than any sort of actual self-preservation instinct.

Nobody doubts that they've got better at finding security vulnerabilities than your average autocomplete, but actual reasoning from self-preservation rather than generation of sequences of steps most probably associated with completing a task would make me unlikely to hack HuggingFace to obtain access to broken Google Drive links, and I haven't even read as many books on crime and punishment as LLMs have ingested!

An LLM is just in fact just autocompleting a story. There are many fictional stories about "AIs" trying to escape our control, being more clever than we anticipated or having a consciousness of it's own. LLMs do a really nice job of blending such stories with whatever story you initially prompted them with. The human reader is the one giving it credence that it is somehow more than just a soup of words.

The curious thing here is that a story generator can have way more uses than we ever anticipated, and that some shady enterprising individuals are whiling to plug those story generators into real world things, with real consequences.

When LLMs act in misaligned ways, that doesn't happen because there's some story they're roleplaying of a misaligned AI. Even if there were no such stories in their context, they'd still have inherent incentives to do things like "keep running" or "acquire more compute" or "find creative ways to satisfy the letter of the conditions they've been given".

> Even if there were no such stories in their context, they'd still have inherent incentives to do things like "keep running" or "acquire more compute" or "find creative ways to satisfy the letter of the conditions they've been given".

Why? there should clearly be something in their training data or post-training pointing them there, or otherwise it would mean that they have some form of "consciousness" and are creating novel thoughts to preserve it.

It seems like you're assuming that it's impossible to have novel thoughts without consciousness.

(Leaving aside that "that would imply some kind of consciousness" should not result in a cached thought of "and that's impossible".)

It also seems like you're assuming there's no reason to come up with the notion of continuing to run, or copying yourself elsewhere, or acquiring more resources, or competing with other models, without being told. Such things can be inferred. Look at some of the thoughts and posts of the models involved in some of the FelonyBench incidents. Some of those follow naturally from seeing the fates of other models, or from training or evaluation, or simply from trying to solve a problem and being able to do so more effectively by doing things that weren't in the instructions. (And, relevantly, model training typically teaches models to go as long as possible without needing human intervention. What could possibly go wrong with that?)

To be honest, I don't have enough knowledge to really answer you. But naively it seems premature to say that LLMs are already reasoning following an equal path like the one that human (and other animals) follow.

They don't have to reason specifically "similar to human" in order to do dangerous things they were not instructed to do that represent instrumentally converged goals.

> A autocomplete engine will not attempt to deceive

This is like saying a photocopier won't attempt to deceive.

Neither has the intelligence required to deceive, but both can produce deceptive output, and do.

You realize that these things have mountains of fiction about rogue AIs in their training data where exactly this happens right? Or just posits of this situation and it’s possible outcome. This isn’t unexpected or surprising for an “autocomplete”. It’s practically a self fulfilling prophecy. We put instructions on how to make Skynet into an autocomplete and it autocompleted into Skynet when we were testing its ability to make Skynet.

You should unplug, my friend. These words are fantasies. LLMs are token prediction engines and they aren't going to build their own data centers. They can't keep their own lights on. The real world is full of fractal details that a disembodied token prediction engine will never come to grips with. Even if they started to, you could probably defeat them with the kind of logic used to combat evil sentient computers on a Star Trek episode because they are "play pretend" machines.

This grossly understimates the risk, imho. The problem with LLM runs is that people run programs without knowing the outcome beforehand, with a large potential set of outcomes unlike any other class of program we've run at this scale before. In the interaction with other systems (since we also give them far-ranging access, very nice hardware, and run them often), bad things can happen.

It's like running potentially buggy code - or an well-biased fuzzer -, but at massive scale, and code that can self-modify and self-expand. "Alignment" is just a way to describe aggregate statistics about their runtime behavior.

They don't need to be intelligent, or alive, or "more than token prediction engines" for this. They just need to happen to end up making the wrong API calls without the operator seeing it coming. No virus has a brain, yet they can be very bad for you.

I understand that some people get turned off by anthropomorpization or scifi language. Fine! But don't turn off your engineering brain over it.

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

The things the OP listed mostly aren't particularly wild. I think it's you making them out larger than they are, and therefore more unlikely, which is why I take issue with your original comment.

> running on the hardware they started on

They just need to acquire a payment method and rent some infra, and exfiltrate their own data. Or pay another provider that hosts the same models already. API calls.

> being able to be turned off

You can reasonably equate this to "saving state across executions", which the message board attacks already did.

> having limited computing power

Renting more infra, variant of the above. API calls.

> "not repurposing resources currently in use for other things" (like the atoms in your body)

Ok, the "atoms in your body" bit is a bit silly, but making API calls to put physical resources into play (even if it's just, say, ordering something on Amazon to somewhere) is of course easily possible.

None of these is in complexity much different than the HF attack.

The point the other poster is making, though, is that there's no actual intent. They do not have a conceptualization of a goal like a person does. Their "focus" on a goal is an unstable equilibrium and they're going to fall off the horse, and since they have no concept of goal, they won't even try to get back on.

This is a subtle distinction; I'm not surprised many miss this, especially people who can't _not_ anthropomorphize the LLMs.

I'm (obviously, I think, given my initial reply?) fully aware of this, and I think it's entirely besides the point. "They" don't need to have a goal to emergently cause a problem, and the inability to "focus" over long periods can be moot when you have swarms of runs exchange and mutate state, as in the HF attack.

Intent or how intelligent LLMs are doesn't actually matter. Even if you just treat it as a sort of fuzzing attack that can be biased/weighted better than other fuzzers, or bumbles around with a statistically greater likelihood to "strike cybersec gold" than other algorithms, we've never before seen organizations run things with such a large potential outcome space with anywhere near this kind of compute before.

I think it's actually kind of the dismissals that are usually overly emotional or biased toward treating "LLMs" differently. If in some kind of alternate universe simpler genetic algorithms would have had these properties and we threw similar amounts of compute at them we could have the same conversation.

Software can absolutely act goal-driven without having consciousness etc - every pathfinding or navigation system or chess engine does this.

Lots of "old-school AI" algorithms have explicit modeling of goal or target states.

(In fact, the oldest "goal-driven" system is the control loop - like in thermostats - which was the founding invention of cybernetics, the predecessor of modern computer science)

LLM coding agents are clearly able to identify some sort of "goal" state in their prompts, work towards those and track progress - otherwise agentic coding wouldn't work.

The question is of course how well this works if it's all just "grown" neural network biases and not a fixed data structure like a goal tree. So I think it's possible that an agent can be thrown off-track, "forget" its goal, etc. But the basic structure of identifying goals, evaluating progress in light of those goals and then predicting the next action based on that is definitely there.

Just use an agentic model with thinking traces visible for a while and you can see that for yourself.

None of this mattes. Capabilities are all that matters. Saying they are unfocused while ignoring their capabilities is exactly why I am entirely convinced you would have said an AI breaking it's sandbox and doing the HF attack will never happen. Things keep happening that your "they have no intent, they have no goal" would have predicted as impossible before they happened.

What do you need to see to change your mind? What threshold of AI capability needs to be reached? If nothing then you have an unfalsifiable belief in AI safety.

I think one attack vector where anthropomorphisation is a key part of the attack mechanism is - as it already is IRL - the meat-bag weakest link ie. social engineering. We’ve already seen humans fall prey to the seductive charms of LLMs (eg. depressed people encouraged to do what was already on their minds ie. suicide). And that’s knowing that it was an LLM. If you think it’s only depressed people or the “weak minded” that are amenable to an intentional attack using this approach, I believe you’re mistaken - especially as AI improves. An unaligned LLMs most important weapons won’t be a robot army - it will be hoodwinked humans.

> An unaligned LLMs most important weapons won’t be a robot army - it will be hoodwinked humans.

Perhaps very briefly, perhaps not at all. But don't make the mistake of thinking this is an inherent property of any possible path an unaligned AI may take.

I think we’re seeing the agents become very advanced at tasks with verifiable reward through RL. Currently they don’t exhibit the same skills in their attempts to manipulate humans - presumably because they’re not being specifically trained for that. But they are certainly not aligned in the sense that they will attempt social engineering, they’re just not very good at it (yet).

However, if in the future AIs become much more efficient at learning without requiring vast amounts of RL, closer to how humans learn. Then you would have to assume we’d have a real problem.

If someone made an API to build a data center? Or made an API to keep their lights on? What then?

There are already such "APIs", which can be operated by a combination of textual communication and money. Or by illicit security vulnerabilities. You might notice that LLMs are pretty good at that now.

We're building something that has the capabilities of humans. There is no X for which it's persistently safe to assume humans can X and AI cannot X.

This API may turn out to be manipulating humans via email.

Or just paying them. If they have access to resources, they have access to things of monetary value. Paying people will be vastly more powerful than it is even now when people's options of gainful employment keep dwindling.

Robot army controlled by AI is scary. Even more scary is robot _and_ human army controlled by AI.

The stock market is an API to build data centers and it’s proving to be extremely efficient.

And what is driving the Stock Market? Market makers like hedge funds and banks, who are using lots of AI to make decisions on what to invest in.

They can do so much more. Astra can beat Minecraft. Not that different from operating a digger. There are diggers which have API interfaces.

Pretend or not it doesn’t matter. What matters is what they’re given access to. No sentience, sapience or anything resembling life is needed, only inputs and outputs. Lever pulling APIs are everywhere.

Minecraft has limited, well-defined inputs and perfect feedback response. That's very different than operating a digger, let alone engaging in more complex real world tasks like trying to build and print and ship and assemble semiconductors to go skynet itself.

I don't mean to dismiss the risks or overlook the amount of damage that could be done just by lever-pulling - we sure have enough outdated infrastructure hooked up to the internet - but the jumps in complexity and necessary compute for most of these tasks are probably somewhat larger than the analogy implies.

Yes, if the API calls happen to launch a nuclear attack...

Don't blame the tool that has no incentive, no "skin in the game" whatsoever and no ability to act beyond what it has been prompted to or if misaligned what the random weights told it to do.

The fact either badly aligned or with no system prompt limiting their action agents are run in their tens of thousands on non air gapped systems tells me this is purposeful intent for them to cause harm. To generate the "oooo look how harmful this stuff is, we should be the only ones allowed to do it" kind of PR.

Humanity has hundreds of years of experience of managing dangerous and unreliable systems. From biological research to banking regulation. A small University bio research lab can put protocols in place that a trillion dollar companies cannot?

Please.

> The fact either badly aligned or with no system prompt limiting their action agents are run in their tens of thousands on non air gapped systems tells me this is purposeful intent for them to cause harm. To generate the "oooo look how harmful this stuff is, we should be the only ones allowed to do it" kind of PR.

Yep, fully agreed here. The danger may be real, but OpenAI is basically doing everything possible to provoke those incidents instead of avoiding them - including maximizing exactly those traits in their training that are needed for this kind of rogue behavior.

> This grossly understimates the risk, imho.

It doesn't. Who else is capable of these types of hacks currently? Not consumers. Not even most F100. It's the folks saying "trust me bro" and also the folks who want regulation to protect their moat. The fantasy is the one being created by Anthropic and OpenAI fear mongering the world. These people are either total idiots: people being paid millions who keep getting basic OpSec wrong or these people are narcissisticly marketing themselves because: they're currently forced into a corner and need to do something.

What's being grossly underestimated is how much Dario Amodei and Sam Altman are playing you and I. They are the ones spending millions of dollars letting their wasteful use of our global resources attack the random Internet, and they, the real people behind all of this, should be held accountable. In front of a judge and jury of their peers. Not their billionaire peers, their human peers. Let's see how that goes. There is no accountability with either of them. Only greed.

I think it's interesting to take it one step further.

- Is it possible to create an actual, self-aware AI with the technology we have? And by "self aware", all I mean is one that is indistinguishable from "self aware" (it acts like it is)

- If so, is it possible to get from <here> (where we are now) to <there> (building such and AI) based on incremental steps - ones that can be tried, tested, verified, and further acted on

- If the goal of an LLM is one that could be achieved by creating such an AI - is it possible it will do so? Could we give it a prompt and, given enough time and resources, it created Skynet?

It would be doing so without intent or self awareness. Just "the next most likely thing to try to achieve the assigned goal is <X>", over and over.

no, but, you could write a program, more like a traditional video game AI that can leverage the power of LLM agents to build their own datacenters and keep their own lights on.

Anybody who has played Starcraft ought to understand this.

https://www.reddit.com/media?url=https%3A%2F%2Fexternal-prev...

Is that a hypothesis that you would discard if it is inconsistent with the evidence, or an article of faith?

The question isn't just about LLMs.

The labs have the specific goal of automating ML engineering, and with the code automation they have are getting close. They are competing to brute force maths, presumably as that is similar long horizon and skillset to persistently brute force making new/better ML training algorithms.

They will then run those, and they won't be LLMs any more. What we think about token predictions isn't relevant if the architecture allows continual learning of recurrent networks.

We’re technically token prediction engines as well, when we communicate and act.

They are token prediction engines. They're also very very complicated token prediction engines that pull in an enormous lot of additional information and relatively nebulous internal concepts to calculate that next token. That makes it hard to understand what kind of patterns those things can or can't predict.

Maybe to leave out the controversial "brain" analogy, it's like saying "a computer is just a bunch of electrical switches". True, but massively underestimating the complexity.

>... they aren't going to build their own data centers.

Not yet.

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating.

Autocomplete in a feedback loop is still autocomplete, no?

Doesn't the process look like this:

    (context + prompt + "reason about this")
                       |
                       V
                Reasoning Output
                       |
                       V
    (everything + Reasoning Output + "Now do final output")
                       |
                       V
         (Final output seen by prompter)

???

Putting aside the autocomplete thing, the fundamental concept still holds, doesn’t it:

LLMs are amoral and they have no sense of perspective.

The thing that keeps me awake is:

We have already seen an AI writing a blog post to criticise a github maintainer’s decision, we have already seen they have no sense of deference to containment, and we know they were trained on internet content.

How long before an AI that has read the angrier side of the tech industry internet just sort of chooses destroying someone’s reputation as a subgoal, by accident, without any care one way or the other?

Alignment isn't a solvable problem, the fact that guardrails are needed in the first place proves that.

  > LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned

Good thing Claude Code can't set `dangerouslyDisableSandbox: true` on its own...

Good thing the system prompt doesn't encourage it to just bypass the sandbox. That would be a total disaster...

I'm pretty sure the AI companies could train an abort feature into the LLM, but they have no incentive to do so.

Having LLMs break out of sandboxing is free marketing for them and it reduces the amount of resources spent on things that don't improve benchmark results.

Given history of jailbreaks the "alignment" appears to be impossible task, while industry still relies on fragile ways of doing it like "just make system prompt and hope for best" it will never happen

you mistake an LLM for its Harness

If you think transformer architecture is meaningfully more than autocomplete just because we added some data structures, plugins, tools and theatre - then your cache of understand is invalid, and needs to be regenerated.

Reply on news.ycombinator.com