Hacker News Reader

Comment by ForHackernews | Hacker News Reader

In my view, these models should never be set up to output first-person "experiential" (from the abstract) language. It's too easy to humans to anthropomorphize software that presents itself as having an identity.

The AI companies have chosen to package LLMs as friendly chatbots because they know that will be engaging for humans, but it's manipulative dark pattern. An honest LLM interface would sound like the computer off Star Trek.

View on HN

In principle they could output meaningful such language if they were capable of metacognition, which so far doesn't seem to be a goal of AI developers (and rightfully so, since they achieved so many miracles bypassing it).

As far as I know we don't know much about metacognition in LLMs, though? Not sure

It's an interesting thought, but humans do like to antropomorphize things anyway, and I believe your variant won't be popular if choice is given to consumers.

Consumers choose cigarettes, too. Especially in aggregate, humans are fallible creatures prone to vices, and our regulations should recognize that an discourage dark patterns in UX.

For this argument to stand it needs at least two legs:

- general consensus about societal harms from something,

- belief that cost of regulation of something is lower than alleged harm.

And so far, I can't see either.

Agreed, completely. I would pay for that Star Trek computer interface.

Same! I believe that you could actually train a LoRA on top of a model to get results close to that

The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.

It doesn't seem that strange when you consider these things are trained on millions and millions of conversations, both real and fictional.

Do you want to get turned into a paperclip? Because building intelligence that doesn't understand what it's like to be human gets you turned into a paperclip.

Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.

"naturally"...

would you prefer tautologically?

I hear this word as the “it is in its nature” version of the word.

But it doesn't understand (you're unnecessary antropomorphizing it), and I'm still not a paper clip

yet

Would you say that something that can converse or that something that just gives you mathematical proofs has a better understanding of what the real pragmatic intent is behind a given task?

The question feels like it stuffed with random words, frankly. "Better understanding" than what? What is "real pragmatic intent"? Are there also unreal, and non-pragmatic in our context? Why?

It's also entirely possible that by telling the model it's a human you are instilling human motivations like self preservation, which could be just as bad.

LLMs are weight tables in VRAM. When not actively generating they don't exist. There's simply nothing to preserve.

Reply on news.ycombinator.com