Hacker News Reader

Comment by lsy | Hacker News Reader

It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios, get shocked that, e.g., feeding emails into a language model results in the emails coming back out, and then write about it with such a seemingly calculated abuse of anthropomorphic language that it completely confuses the basic issues at stake with these models. I understand that the media laps this stuff up so Anthropic probably encourages it internally (or seem to be, based on their recent publications) but don’t researchers want to be accurate and precise here?

View on HN

When we use LLMs as agents, this errant behaviour matters - regardless of whether it comes from sci-fi “emergent sentience” or just autocomplete of the training data. It puts a soft constraint on how we can use agentic autocomplete.

Agree the media is having a field day with this and a lot of people will draw bad conclusions about it being sentient etc.

But I think the thing that needs to be communicated effectively is that these these “agentic” systems could cause serious havoc if people give them too much control.

If an LLM decides to blackmail an engineer in service of some goal or preference that has arisen from its training data or instructions, and actually has the ability to follow through (bc people are stupid enough to cede control to these systems), that’s really bad news.

Saying “it’s just doing autocomplete!” totally misses the point.

i am sure plenty of bad things are waiting to be discovered

https://www.pillar.security/blog/new-vulnerability-in-github...

That statement is as true now as it was when some caveperson invented fire.

It's a massive hype bubble unrivaled in scale by anything that has ever come before it, so all the AI providers have huge vested interests in making it seem like these systems are "sentient". All of the marketing is riddled with anthropomorphization (is that a word?). "It's like a Junior!", "It's like your secretary!", "But humans also do X!" etc.

The other day on the Claude 4 announcement post [1], people were talking about Claude "threatening people" that wanted to shut it down or whatever. It's absolute lunacy, OpenAI did the same with GPT 2, and now the Claude team is doing the exact same idiotic marketing stunts and people are still somehow falling for it.

[1] https://news.ycombinator.com/item?id=44065616

Wow… you weren’t around for the dot bomb era?

This is just dotcom 2.0, except now people are throwing Billions into every single idiotic idea out there. There's fucking toothbrushes with "AI" functionality now.

It was billions back then too. There were internet connected toothbrushes. Hundreds of millions of dollars went to completely idiotic startups.

It’s just a good rush. It’s happened before, it will happen again. It’s not even irrational; trillions of dollars were made by the dot com era companies that did succeed. I have no doubt AI will be the same.

But since nobody knows who will be successful and there’s tons of money sloshing around, a lot of / most of it is going to be wasted in totally predictable ways.

> It’s honestly a little discouraging to me that the state of “research” here is to make up sci fi scenarios,

it's not research, it's marketing

the main aim of this "research" is making sure that you focus on this absurd risk, and not on the real risk: the inherent and unfixable unreliability of these systems

what they want is journalists to read through the "system card", spot this tripe and produce articles with titles like "Claude 4 is close to becoming Skynet"

they then get billions of free publicity, and a never ending source braindead investors with buckets of money

additionally: it worries clueless CEOs, who then rush to introduce AI internally in fear of being competed out of business by other sloppers

these systems are dangerous because of their inherent unreliability, they will cause untold damage if they end up in control systems

but the blackmail is simply parroting some fiction that was in its training set

Reply on news.ycombinator.com