Hacker News Reader

Comment by gfody | Hacker News Reader

calling machine-learned human behavior an "alien mind" that we must "teach how to love" is feeling very off to me. it's misleading in a way that feels dishonest, like don't think about where the behavior came from marvel at it and fear it instead.

View on HN

The most charitable way I can describe it is just extremely low quality sci-fi fan fiction. I think that's too charitable, because I believe it's far more cynical than that. They're deliberately playing into these sort of techno-religious beliefs that have taken root in the wake of Kurzweil, et al., fanned by LLM psychosis, influencer marketing, and a deluge of this kind of sci-fi marketing copy. It's just chatbots, guys. Relax.

Alien mind is in my view the best mental model - LLMs are not merely stochastic parrots, are not like humans, are not like animals.

They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.

I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.

pretending that it's something like a mind at all is what's misleading. it's more like a cast of a bunch of overlapping/entangled thinkprints, and pushing activation through it produces new prints. it can already "love" because that behaviors in the data along with hate and everything else.

acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources

the behaviour is alien because it is different from human, they don't know in detail how those trillions thinkprints work together to produce output

openai "knew" about scaling law for a decade now and still can't fully explain it, why you think they are dishonest here?

it's a rampant misconception that something needs to be mind-like to make new prints from the cast. scaling up produced higher fidelity prints from a higher resolution cast. not knowing in detail why activation heads work as well as they do is besides the point when we are clearly reproducing the human behavior that created the prints in the first place - in other words our behavior, plagiarized at scale - shrugging about how mysterious that is while committing hundreds of millions to snatch up more personal data sources is a little dishonest

it feels unhinged and makes me think we should just put every engineer working at these labs in jail to pause this shit until we can figure out what the fuck they are doing over there

Reply on news.ycombinator.com