Hacker News Reader

Comment by aardvarkr | Hacker News Reader

“Intelligence” implies that actual thought is happening. These language models are amazing at token prediction but it’s a mistake to believe that they’re actually intelligent.

Give it an inane prompt to better see the difference, like "write me an essay about camels and their value in cybersecurity" which gives us the answer

"The camel is a valuable animal in the cybersecurity field due to its ability to store large amounts of data in its hump. Camels are able to cross vast distances with little water, which makes them ideal for carrying large amounts of data across networks. They are also very sturdy animals, able to withstand harsh conditions and even attacks from cyber criminals. In addition, their long eyelashes protect their eyes from sand and dust, making them perfect for working in dusty environments such as data centers."

View on HN

That looks a lot like ‘playfulness’, which would generally be considered a sign of intelligence if it were observed in, say, animals.

You asked it to write an essay on a nonsensical premise so you got a nonsensical essay. If you just ask chatGPT ‘what is the value of camels in cybersecurity’ it doesn’t see that as a cue to play:

> I'm not sure what you mean by "the value of camels in cybersecurity." Camels are a type of mammal that are commonly used for transportation and as a source of food and other resources in certain parts of the world. They do not have any direct connection to cybersecurity.

That’s a very generous interpretation. It’s not designed to give “playful” responses and instead what you’re seeing is a system so eager to answer the question that it doesn’t care how wrong it is.

A human answer would say camels have very little to do with data security. A Google search would show you a link to Apache Camel. ChatGPT tells you how amazing a camel’s humps are for data storage.

You bring up a great point that how you ask the question changes how the system responds. An intelligent being such as yourself would see the two as asking the same thing.

> That’s a very generous interpretation.

Otherwise known as a "sense of humor".

> A human answer would say camels have very little to do with data security.

Correction, a human without a sense of humor would say that camels have very little to do with data security.

Edit: If I'm getting downvotes, I might as well lean into them and make it worth my while...

Otherwise known as a "sense of humor". Something that ChatGPT seems to have that you don't. Oh, snap!

Can you provide an example of ChatGPT laughing at a joke? It’s fascinating that you go to such lengths to insist that it has human characteristics that it clearly does not possess.

Dude...

It's not designed to give correct answers either. It's just designed to produce text output that's similar to the text in the training set.

it's painfully obvious too. one of gpt's most noticeable things is that it sounds like a high schooler trying to fill up content for a 5 paragraph essay

If ChatGPT isn't designed to give playful responses, and yet it frequently does produces playful responses (do you think a serious, fact oriented system would happily write an explanation of the A-star algorithm in iambic pentameter?), I'm even more impressed. If the playfulness is an emergent property of the system, I don't think that argues against the idea that it's a product of some kind of intelligence.

But in fact I suspect that ChatGPT was designed to be playful, as well as helpful. I think the human based reinforcement phase they put it through to tune the chat output was probably used to encourage some of these creative generative responses where it takes a ludicrous idea and proceeds to execute on it.

Back in the 60s or 70s, it seemed a psychologist was finding solid evidence for telepathy, by having subjects guess the symbol showing on a card being viewed by someone out of the room. It eventually turned out that he was doing things like finding some runs in the data where the subject was not naming the current symbol, but instead the prior or subsequent one better than chance, and counting those as successes. It seems to me that an ex post facto attribution of playfulness to a result that does not show knowledge is essentially the same.

ChatGPT was not designed to be either playful or helpful; it was designed to pick a likely next word, and repeat.

"ChatGPT was not designed to be either playful or helpful"

You don't think OpenAI thought putting it out for public access might serve some sort of purpose? They just created a technology based on an arbitrary algorithm without any thought as to what it might actually produce?

They designed it to do what I said (which is far from being an arbitrary algorithm), and then put it out there to find out what that would result in. Ex post facto anthropomorphizing about what some cherry-picked samples of output looks like does not even establish that it is doing those things, let alone that it was designed to do them.

> It’s not designed to give “playful” responses and instead what you’re seeing is a system so eager to answer the question that it doesn’t care how wrong it is.

It absolutely is trained on loads of "playful" data

It is trained on "playful" and non-playful data indiscriminately.

Poe's Law and all that jazz...

I learned something new today, thanks!

That's a pretty funny response to a pretty funny question!

I'm on your team, here, Dot. Rather playful response, like someone would write in response to a creative writing prompt. In fact, I think this is the correct way to answer that question.

Found the AI!

Because, until now, only humans could generate language like this, there is a strong temptation to anthropomorphize and attribute intent.

Reply on news.ycombinator.com