Hacker News Reader

Comment by aqme28 | Hacker News Reader

> ChatGPT is a glorified word predictor. It isn’t sentient. It doesn’t know what it’s saying, and yes, you can coax it into admitting that it wants to take over the world or saying hurtful things (although it was specially conditioned during training to try to suppress such output). It’s simply stringing words together using an expansive statistical model built from billions of sentences.

How do you differentiate it from the human mind? Do we understand ourselves well enough to say that we aren’t also just self-reflective reinforcement learners doing statistical inference on a library of all our “training data”?

View on HN

Somewhat related to this:

We seem to operate on the assumption that sentience is "better," but I'm not sure that's something we can demonstrate anyway.

At some point, given sufficient training data, it's entirely possible that a model which "doesn't know what it's saying" and is "stringing words together using an expansive statistical model" will outperform a human at the vast, vast majority of tasks we need. AI that is better at 95% of the work done today, but struggles at the 5% that perhaps does truly require "sentience" is still a terrifying new reality.

In fact, it's approximately how humans use animals today. We're really great at a lot of things, but dogs can certainly smell better than we can. Turns out, we don't need to have the best nose on the planet to be the dominant species here.

We can't even prove other people are sentient. This is not a fruitful line of inquiry.

If this is a reply to me, I think you missed the point I'm making here. I don't care if we can prove other people are sentient or not.

My point is that it may well not matter whether a thing is sentient or not if a well-trained algorithm can achieve the same or better results as something that we believe is sentient.

You say it may not matter, I say I think it certainly doesn't.

Well for a start the human mind involves a series of chemical reactions optimised by evolutionary wiring and physical world interaction towards self replication, so when a human says "I feel horny" there's a whole bunch of stuff going on in there that there's no reason to suspect is replicated in a neural network optimised for text transformation.

When a silicon based hardware computes that as a response, it isn't because a whole bunch of chemical reactions is making it desire particular sensations and hormonal responses, but because the limited amount of information on human horniness conveyed as text strings implies it's a high probability continuation to its input (probably because someone forgot to censor the training set...)

Insisting comparable outputs make the two are fundamentally the same isn't so much taking the human mind off a pedestal as putting a subset of i/o that pleases the human mind on a pedestal and arguing nothing else in the world makes any material difference.

“One is chemical and one is in silicon” doesn’t strike me as a very meaningful distinction. Why does that really matter?

A computer simulation of water can easily convince the human eye it's water, both in terms of pixel perfect representation and simulated behaviour in simulated environments. Until they try to put it in a bottle and drink it.

Turns out that physics of what it actually is matters more than human observation that some of the pretty output patterns look identical or superior to the real thing.

(And aside from being physically very dissimilar, stuff like even attempting to model human sex drive is entirely superfluous to an LLM's ability to mimic human sexy talk, so we can safely assume that it isn't actually horny just because it's successfully catfishing us!)

I've interacted with many people online, only through text, and my life has significantly changed because of many of those interactions. The effect on my life would have been the same whether the entities typing were made out of silicon or carbon.

Sure, and I've been deeply affected by books, but I'm not going to start using that as a basis for an argument a book and a human think in exactly the same way

This was in response to your comment about how you can tell that a water simulation is fake by trying to dip a water bottle in there. The distinction between chemical and silicon doesn't matter when the output is text. There's no physical test you can perform in the text, like dipping a water bottle in water, to see if it's chemical or silicon.

If you test both on the same terms (i.e. only interaction via a remote terminal) then a decent simulation can entirely convince humans that a bottle has been dipped in it and water removed from it too. But it still doesn't have the essential properties of H20, it just looks like it in many ways to some imperfect observers.

Testing is a moot point when my original argument was that it there is no reason to assume that a converts-to-ASCII subset of i/o as it is perceived by a [remote] human observer other is the only differences between two dissimilar physical processes (one of which we know results in sensory experiences, self awareness etc). Takes a lot more belief that the human mind is special to believe that sensory experience etc resides not in physics but whether human observation deduces the entity has sensory experience.

This kind of thought experiment always reminds me of Measure of a Man from Star Trek TNG.

It shouldn't really...

Measure of a man was about social issues surrounding agi if we assume a perfect agi exists, but the only thing agi and language models have in common is a marketing department.

Plus plenty of people just string words together, yet cannot answer anything remotely structured like a simple program or even simple arithmetic. Yet they get the sentient label.

Human mind can perform actual reasoning, while ChatGPT only mirrors the output of reasoning and when it gets output correctly it's due to mixture of luck and closeness to training material.

Human mind or even something like Wolfram Alpha can perform reasoning.

Can you explain your proof of that?

Not OP, but basically:

Humans have the capacity to come up with new language, new ideas, and basically everything in our human world was made up by someone.

ChatPT or similar, without any training data, cannot do this. Thus they're simply imitating

Humans are not trained? How much of training is responsible for humans being able to come up with new language and new ideas?

Thats assuming modern humans, I was talking about ancient humans, before civilisation. You could argue thats where the creative mind shows up most, as there are very few humans to imitate.

What experiment can you do to confirm this? If I ask ChatGPT to come up with a new language, it will do it. How do I distinguish that from what a human comes up with?

By not giving them any examples of language. I would expect humans to come up with a language, if not vocal, without guidance. I doubt GPT would do anything without training data to imitate.

Humans can't either, without training data. The biggest difference between chatGPT and humans is that humans are not trained solely on language.

I think this is going to change very soon.

Based on the current advances, in about a year we should see the first real-world interaction robot that learns from its environment (probably Tesla or OpenAI).

I'm curious (just leaving it here to see what happens in the future), what will be the excuse of Google this time.

This is again the same situation: Google has supposedly superior tech but not releasing it (or maybe it's as good as Bard...)

The important difference is that humans are trained on a lot less data than ChatGPT. This implies that the human brain and LLMs are very different, the human brain likely has a lot of language faculties pre-encoded (this is the main argument of Universal Grammar). OpenAI's GPT 4 is now trained on visual data.

Anyway, I think a lot of ongoing conversations have orthogonal arguments. ChatGPT can be both impressive and generate topics broader than the average human while not giving us deeper insight into how human language works.

ChatGPT and similar do seem to make new things, arguably they do it more freely than the average adult human.

Art generators are the most obvious example to me. They regularly create depictions of entirely new animals that may look like a combination of known species.

People got a kick out of art AIs struggling to include words as we recognize them. How can we say what looked like gibberish to us wasn't actually part of a language the AI invented as part of the art piece, like Tolkien inventing elvish for a book?

Humans require training data as well.

And what do you think of the Mark Twain quote:

“ There is no such thing as a new idea. It is impossible. We simply take a lot of old ideas and put them into a sort of mental kaleidoscope. We give them a turn and they make new and curious combinations. We keep on turning and making new combinations indefinitely; but they are the same old pieces of colored glass that have been in use through all the ages.”

I’d argue ChatGPT can indeed be creative, as it can combine ideas in new ways.

You could argue like that against anything

Plenty of examples of it coming up with new languages or ideas. And it’s very hard for a person to come up with a new language completely independent of reference to other known languages.

Just try to talk with it about math. You'll quickly see that it's as if you talked to a person who doesn't understand anything about math. Just read some books about it and attempts to mimic their style to appear to be smart and knowledgable.

Without any external plugins, GPT can encode and decode base64 strings that are totally new. Again "luck" ?

If a system is so lucky that it gives you the right answer 9 times out of 10, it's perhaps not luck anymore.

It cannot, encode base64 it only remember, see this conversation:

https://news.ycombinator.com/item?id=34322223

It totally can (try it if you don't believe it).

In your message you say it is gibberish, but I have completely different results and get very good Base64 on super long and random strings.

I frequently use Base64 (both ways) to bypass filters in both GPT-3 and 4/Bing so I'm sure it works ;)

It sometimes make very small mistakes but overall amazing.

At this stage if it can work on random data that never appeared in the training set it's not just luck, it means it has acquired that skill and learnt how to generalise it.

Did you tried longer sentence and not singles words? Did you also read the conversation?

Edit: ok it looks like it can now convert in base64, I'm sure it couldn't when I tested 2 months ago.

It could when I tested 1 week after the first version of chatGPT was in private beta. It's always been able to convert base64 both ways.

It sometimes gets some of the conversion wrong or converts a related word instead of the word you actually asked it to convert. This strongly suggests that it's the actual LLM doing the conversion (and there's no reason to believe it wouldn't be).

This behavior will likely be replicated in open source LLMs soon.

Ask it to “reason through” a problem and then ask it to give you an answer. How’s that different from thinking?

It's just wrong. That's how you can tell. Actual reasoning leads to sensible conclusions.

Coming to the wrong conclusion doesn’t mean I wasn’t thinking through the problem.

It definitely means that it was thinking wrongly if at all. Just talk to GPT about math. You'll quickly change your mind about the possibility of it thinking.

LLMs are bad at arithmetic due to tokenization limitations but they're actually pretty decent at mathematical reasoning. You don't know what you're talking about I'm afraid.

https://www.lesswrong.com/posts/qy5dF7bQcFjSKaW58/bad-at-ari...

Please just try. It's horrible at mathematical reasoning. Use just words to avoid problems with tokenization. Alternatively just read through the link you provided. It has many examples of failures of GPT and gabage it produces when talked to about math.

Clearly you haven't read it then.

The provided example directly shows ability in mathematical reasoning by coming up with a novel concept and example case, it is just poor in arithmetic.

Math is not simply arithmetic abilities, you seem unable to comprehend this.

Sure, sure.

"I want it to come up with a new idea. Its first attempt was to just regurgitate the definition of the set of zero-divisors (a very basic concept), and (falsely) asserted that they formed an ideal (among other false claims about endomorphism rings)."

"I tried a few more times, and it gave a few more examples of ideas that are well-known in ring theory (with a few less-than-true modifications sometimes), insisting that they are new and original."

"This in particular is quite an interesting failure. "

"So there we have it. A new definition. One example (of a 4-cohesive ring) extracted with only mild handholding, and another example (of a 2-cohesive ideal) extracted by cherry-picking, error-forgiveness, and some more serious handholding."

"Some errors (being bad at arithmetic) will almost certainly be fixed in the fairly near future." - and this opinion is based on absolutely nothing.

When a model "reasons through" a problem its just outputting text that is statistically likely to appear in the context of "reasoning through" things. There is no intent, consideration of the options available, the implications, possible outcomes.

However, the result often looks the same, which is neat

"thinking" and reasoning can be done by toddlers with a dataset a fraction of a fraction of the size that even the simplest language models are trained on.

I don't understand this thinking that it's x because it looks like x(thinking, artistic creativity, etc.). I can prompt Google for incrementally more correct answers to a problem, does that mean there's no difference between "google" and "thought"?

Re: word predictor, there is a interesting experiment: tell it to skip every other letter in evrry word, for example you ask it "hw ae yu?" and it answers flawlessly. You can tell it to reverse the order of letters or communicate using first letters only. I'm sure the internet doesn't have strange conversations "h a y? im d f" but gpt has figured it out. If you tell it to use a made up numeric language, it will do so easily, and it won't forget to say that the word 652884 is forbidden by its preprompt. And it does all that without internal "thinking loop".

Reply on news.ycombinator.com