On ChatGPT
acoup.blog
acoup.blog
they pretend to be like minds – like human minds. But it is only pretend, there is no mind there and that is the key to understanding what ChatGPT is
And surely they have no mind like us. But this obsession with whether or not it is like us, seems to miss the point, doesn't it. Is it useful? Despite the writers insistence that it is not, my children provide my with many examples that it is.
For example, my daughter is learning to program and got negative feedback on her commenting style. (which admittedly where mostly absent) So, she asked how to do it better. The teacher gave only non-committal responses, like being helpful to the reader, etc. It is not the kind of instruction that is helpful for her. So she gave the whole program the ChatGTP and asked it to insert comments without changing the code. The next day, she took the output to the teacher and asked if this is what he meant. It was perfect. But now, she knows what is expected of her, and now she can do it without ChatGTP.
You can easily test for this by just throwing hard problems at them that they haven't seen before, Google does that in their interviews for example (but many who do similar just takes problems people definitely have seen).
So doing similar kinds of testing on ChatGPT we see that it doesn't understand the information. It can repeat a lot of stuff more or less verbatim, but it cannot apply most of the knowledge it can repeat, so it doesn't understand any of it. The only thing ChatGPT really understands are relationships between words that has been repeated in texts it has seen, that is some level of understanding but that has no relationship to what ChatGPT says about the words.
When you ask ChatGPT to describe something it repeats a description it has seen of the word, when you ask it to do something with a word then it fully ignores that description and instead look at the relationships it has seen the word have to other words. So its understanding and its knowledge are completely separate things, so you can say that it doesn't understand anything it knows.
I agree with this, it's the old "are submarines swimming" kind of question, semantics and not substance, i.e. irrelevant.
However this is an article written by someone non-technical for other non-technical people, and this (emphasis is the author's) is perfectly on-point:
> The tricky part is that ChatGPT and chatbots like it are designed to make use of a very influential human cognitive bias that we all have: the tendency to view things which are not people as people or at least as being like people.
Which is absolutely a problem. Whether or not you ascribe a "mind" to it, laypeople absolutely need to understand that it's not a human they're talking to.
> Is it useful?
A better question is useful to whom and useful how. As a search engine it's pretty miserable for instance, as we found out. I'd like to leave myself some wiggle room because it's way too early to tell, but my hunch is that operating on natural language is a profound limitation that will make it far less useful than it is generally believed.
That's hard when the bot either gets angry at you or declares its love if you spend an hour with it. But is anthropomorphising AIs useful? that's the important question. I think it is useful. If you tell GPT3 it is a respected scientist, it will have higher accuracy solving tasks.
No it isn't. It's disastrous. The bot declares your love for you if you spend an hour with it because it's trained on incel shit from reddit, thinking anything else is 100% a user problem.
> If you tell GPT3 it is a respected scientist, it will have higher accuracy solving tasks.
Even taking that statement at face value, what I think about AI has no bearing on what it does. You are giving input tokens to a machine and you are fooling yourself if you think of it as anything but.
This is exactly the problem about natural language I'm pointing out. I don't want a friend, a coworker or an assistant, I want a mechanical slave to instruct in very specific ways, and natural language is a pretty lousy way to give orders.
This is the first hell yea use case I have seen for chat gtp! Thank you and your daughter!
I've often wondered about this, because this answers the question "are we alone on this planet?" with a clear, resounding "no".
For example, every species has a mind in it's DNA. It communicates very slowly and it thinks and communicates by killing large amounts of life. Or perhaps it can be better put as it communicates by enhancing or diminishing specific forms of life, not always outright killing them. You can talk to this, like we've done accidentally with vaccines. DNA's answer would then be antibiotic resistance in dangerous diseases. It's a very different "mind".
A lot of folks are dismissive of these criticisms by claiming that we have no evidence that human cognition is fundamentally different, but it must be - our referents are based in the real world, whereas the LLMs have no such referents, and it is this tie-back to reality that underpins our ability to act intelligently. And the argument that an equivalent understanding can somehow be an emergent property is undermined by the observation that the LLMs have no concept of truth, and no ability to tell whether their output is pure BS.
I'm not going to analyse particular flaws of separate comments because the article itself is really comprehensive and answers for many objections are better in original form there than any reproduction I can make here. But the high number of rebuttals that obviously have problems either understanding the essay or formulating a solid counterargument is curious. An obvious explanation is that some people couldn't resist the pleasure of letting AI dismiss the statement about insufficient AI capabilities, but that's a lazy one.
Comments saying current AI is comparable to actual human inteligence may be right even though I can clearly see that AI is not performing actions I consider necessary for thinking. It's because I consider my own mind as a model of human inteligence, but as I learned many times before, the thinking process may be hugely different for different people. I have no idea if other people also do need to create an explicit model of something in their head to be able to "think" about it.
It would explain A LOT for me if thinking of some other people really works in the same way as in GPT. That situation should however still be viewed not as a technology enhancement so big it reaches human level but reconsideration of what we consider a human level.
This happens to me a lot. I want to make a joke, but it gets so long an complex that it becomes an analysis.
not that i disagree, but what would you have it change to?
The fact that there's no need to memorize anything any more, means that any tests which allows the use of tech (like the internet, or chatGPT) to retrieve data, means that students no longer need to commit things to long term memory.
However, having the capability, and be able to recall facts when needed, and at high speed, is something that is foundational to higher creative thoughts.
This higher, creative thought, is not really easy to test, so the memorization is the proxy.
I do not know what form education (and the testing of it) would take on, if tools like chatGPT is allowed to be used.
Teachers aren’t interested in a recount of the battle of so-and-so but in training you to gather knowledge, structure your thoughts and express them clearly.
You can only learn that by doing. A chatbot bypasses the learning process, so you will have neither gained subject knowledge nor methodical one.
https://www.nytimes.com/1991/09/29/opinion/the-calculator-cr...
I'm pretty sure that a "sufficiently smart" chatbot (or maybe even an extra dumb one) is a useful tool "in training you to gather knowledge, structure your thoughts, and express them clearly". I've found it remarkably useful for clarifying my thoughts, considering alternative arguments, and general tomfoolery that can spark creativity.
I am mostly worried about young people who will grow up relying too much on ChatGPT, what will they do when they do not have a bot hand-holding them through some complicated idea? And if this kind of bots become so ubiquitous, what is the place for humans?
When calculators became wide-spread, we calculated a lot more. When LLM become wide-spread, we will.. Think more? I seriously don't know.
I certainly benefit greatly already from using LLMs to accomplish a number of tasks. I think the answer on where the responsibility lies depends greatly on your view of the same sorts of questions around auteur theory - is the director responsible for the quality of the film? Or is it the writer of the screenplay? What about the cast, or the producers? Is Microsoft the author if you write a novel in Word, without scribing the lines onto the page yourself? I think it's going to be very interesting to see how all of this plays out, and where the lines are drawn. I suspect that what is causing concern now will, in ten years perhaps, be normal, obvious and not even discussed.
I agree, and that is the issue. People like us can use LLMs effectively because we are already capable of expressing our thoughts in a decent manner and we can recognize when the output does not make sense. But to know whether the results can be trusted or not, you already need to be one level above that. If one is not capable of producing a coherent argument on their own, how can they evaluate whether an argument they hear is itself coherent? And if one, say because of lazyness, relies on LLMs from their childhood to fill in all the difficult steps, how will they learn how to do it on their own? Practising has always been the best way to learn things.
> I think it's going to be very interesting to see how all of this plays out, and where the lines are drawn. I suspect that what is causing concern now will, in ten years perhaps, be normal, obvious and not even discussed.
Well said.
For example, I had to memorize the structural formulas for all amino acids in university. This does seem a bit useless at first, but is actually very important the moment you work with protein sequences or structures. You might not need the exact structure, but if you read about a specific important residue in a protein or a mutation in one you need to understand the properties of the involved amino acids to make sense to this. And if you had to look that up every time you'd never get through a paper.
Being able to distinguish between useful, valid information and whatever YouTube video the search engine happened to turned up is going to be an even more important skill for those kids in school now than it was for those of us who were in college when Google was a hot new startup.
For those of you who didn't read the whole post (and it was long, so I kind of understand), Mr. Devereaux made an aside about his belief in the continuing value of initially learning how to do arithmetic without a calculator despite their easy availability over the past few decades, an opinion I've always shared.
Before reading this post, I still believed the same about the value of learning how to write essays ("delivery boxes for thoughts" was his expression, I think) and will make sure that my kid can write one with just a pencil and paper, even though he'll also be able to use whatever technical assistance is available in 10-15 years. He's learning to draw and make letters with crayons and pens before I'll let him spend a lot of time with my iPad; he's sussed out how that worked just by watching me, so I'm not concerned about a technology gap with his future classmates.
This post gives me something to forward to my non-technical but curious friends when they ask about ChatGPT and similar.
But this is somewhat off topic because the article is talking about essays in the context of a university, not school, education.
(Not to mention that parents are still at least partially responsible for their children's education.)
There is a broader question about whether it is possible to learn from reading. Can a blind person ever really understand “blue”? If not, what can be learnt by reading? Why do we rely on reading and writing so heavily for learning?
Edit: just want to note that this is a great site, really like the author’s approach and style. And maybe if I’ve not learnt from his writing, at the very least it is a good read :-)
I think that the argument for qualia is pretty weak - if we can have a conversation about "blueness" with a blind person, we have to admit the possibility of "understanding" existing in some form in LLMs.
Even if it's just a statistical emulation, at what point does a statistical emulation become reality, if it's a good enough emulation down to the metaphorical planck distance.
How the tables have turned
My understanding is "tech enthousiast" level, so happy to learn.
So for example, the text "123456789" is tokenised as "123", "45", "67", "89", and the actual input to the model would be the token IDs: [10163, 2231, 3134, 4531]. Whereas the text "1234" is tokenised as "12", "34" with IDs [1065, 2682]. So learning how these relate in terms of individual digits is pretty hard, as it never gets to see the individual digits.
I see it analogous to asking a human why they don't just "learn all the answers to simple arithmetic involving integers below 10,000" - you possibly could, it would just be a huge waste of time when you can instead learn the algorithm directly. Of course, LLMs are inherently a layer on top of an existing system which solves those problems quite well already, so it'd be somewhat silly there too.
I consider the letters to correspond to facts about the world, consider the facts symbolically, and then map back to the letters.
It has the same output, but it’s a different process.
In fact now that I think of it, almost the only fields that seem to be using them would be... foreign language teaching ! (Where they are probably appropriate, since so much of it is just pattern memorisation...)
Yes, grading free form writing tasks is a lot more work, but they are also an obviously vastly superior form of exam question.
> Ideally, the essay writer has first observed their subject, then drawn some sort of analytical conclusion about that subject, then organized their evidence in a way that expresses the logical connections between various pieces of evidence, before finally communicating that to a reader in a way that is clear and persuasive.
Of the elements discussed about "writing an essay" above, none are clearly impossible for a large language model to do at the present, despite the author's insistence that some of them are impossible. If the subject is something in writing (and you can pass it in the prompt), or is written about in the training set, then the model has indeed "observed" it, and to get it to draw analytical conclusions all you have to do is ask nicely and correctly, maybe using those words. Again, with organizing evidence, you merely must ask for it by name, rather than hoping that the model understands what you mean when you say "write an essay" - ask for "organize the evidence in a way to support/contradict the conclusion X", and finally of course communicating cogently to the reader is something the writer correctly understands is possible even with a trivial prompt.
Sure, it moves the challenge from "write an essay" to "design a prompt to get the language model to write an essay", but that's "not writing an essay", or is it?
It is immensely frustrating that people feel like a large language model is "cheating", but also don't consider using word processing software with grammar and spelling correction "cheating", even though they use fundamentally the same processes behind the scene and differ only in the user interface, really.
[1] I don't like the term AI, I prefer large language model because it is unambiguous about the properties of the system - AI implies thought processes, which is a distraction at best from the useful properties of a complex statistical model.
i think it's a stretch to claim that spelling/grammar correction and large language models are similar or fundamentally the same.
that's like saying the horse-drawn carriage is fundamentally the same as a plane. Sure, they both move people around, but the differences are much larger than the similarities.
Spellcheck was one of the first language models (albeit miniscule) in widespread use. You can draw a direct line from bloom filters for spellcheck in resource constrained environments to predictive text and thence on to LLMs.
Spellcheckers for the msot part act on what you made. ChatGPT in this case is used to generate stuff, so you don't have to make it.
That's a huge difference and the first analogy is more fitting than the Model T one.
Spellcheckers are more like wheelbarrows, they don't move anything by themselves. ChatGPT is like an airplane with a broken autopilot.
"I have made this letter longer than usual, only because I have not had the time to make it shorter." - Blaise Pascal
But people who think ChatGPT is great usually aren't capable of reading that far :P
For most interesting questions you could write anything between a few sentences and a full book, depending on how much you develop and defend your arguments, evaluate other possible answers, etc.
The word count is a signal on where on that spectrum you should be aiming.
For an advanced student writing about sufficiently complex topic, the challenge is to develop an argument in less than 5000 words, rather than to reach 5000 words.
And there is a big difference between writing an outline (which is valuable and important!) and actually expressing those ideas fully. The latter forces you to clarify your ideas much more precisely - which is a really valuable experience in exploring and understanding the details of a topic.
In that case sure, the whole exercise is kind of a waste of time, but you could say that about basically any educational method surely? You can take a horse to water and all that...
> But it is only pretend, there is no mind there and that is the key to understanding what ChatGPT is (and thus what it is capable of).
This is where an educator thinks they know AI better than people who work on it. A-priori decision it can't work. What they miss when they do that is the special quality of the training set. Language has special properties that lead to the emergence of abilities previously only possible for humans.
Humans for example like to read fiction. It's all pretend, but enjoyable, maybe also instructive. We can simulate other people and their actions, we can infer their emotional states and motives. But it's still all pretend. What if language models can do the same - infer our emotional states, and respond in kind? It would be like generating a novel, something it has seen plenty of in the training data.
The magic dust is not in the model and the next token prediction task, but in what data the model was trained on. Our language, our mind stream, taken from real experiences.
Language has the ability to create things like chatGPT, language can create modern capable humans from just babies. The author missed language and only saw flaws on the model.
Already, by coupling search with language model like bingChat we see even more extreme cases. Now it knows you said things about it on Twitter. These are already going beyond pure LLM. They are going to have a whole host of apps: calculator, calendar, search engine, python and JS execution engines, simulators, recursive calls, language chains, games, memory modules, episodic memory, knowledge bases, etc. I call this paradigm "language model with toys". And they will have this discussion and all the other discussions about it in the next training corpus, thus gaining a way to create a Self. It will have a self because we talk about it like a human.
Citation needed.
> Humans for example like to read fiction. It's all pretend, but enjoyable, maybe also instructive. We can simulate other people and their actions, we can infer their emotional states and motives.
Humans like to read fiction that is extremely rooted in our actual experience of the world. We don't read scifi about 5-dimensional universes with no protagonist that mostly behaves like people. Even on this very blog you'll find excellent articles on why Rings of Powers is bad because it's not believable, and why Lord of the Rings is great because it is (and rooted in much historical knowledge).
> Language has the ability to create things like chatGPT, language can create modern capable humans from just babies.
No? Babies grow into adults after years of experiencing the real world at every instant, as well as the social world. Language starts as a tool to express basic concepts that are very much grounded in the real world, and abstraction comes years later. It's not language that makes babies into adults!
That's going to be really important. Large language models can already summarize documents. How long before you can tell a system "Read this book. Then I will have some questions about it"? Once you have that, you can apply the base model to domain-specific problems.
Sure, these issues will be fixed eventually, but the point of the article is that it can't replace the thinking of a student to write a quality essay. It might be able to regurgitate search results which will need to be fact checked, so at best it's useful as a research tool.
BTW, Bing's AI is a really poor counterexample, as it can't even return search results reliably.
Yesterday I read Wolfram's piece and he wrote:
> I think, is that language is at a fundamental level somehow simpler than it seems.
If Wolfram is true we could argument that GPT-3 was successful in capturing an at least an important aspect of language. And if we also presume that language is part of the mind, then we should refute the statement that there is [absolutely] no mind in GPT-3.
However, GPT-3's mind, if at all, is not human. And this is difficult for us to keep in mind. After all we anthropomorphize everything and their grandsomething.
Even I catch myself having the urge to be polite to ChatGPT and saying thank you. Or, the last time I even told ChatGPT not to worry about something. I know exactly that this is just weird and incongruous, but it did make me feel better, so I just did that. YOLO.
> ChatGPT is, in fact, incapable of knowing anything at all.
All my knowledge about modern high-energy physics comes from reading about them. There are a great many things that I know of only second-hand, and not direct observation. Theoretically, any AI's knowledge of these things cannot be worse than mine, then.
this is foundamentally wrong. I've been instructing chat gpt about wargame rules, and it's understanding those rules well enough to run rounds of combat. that is far beyond what this article pretends chatgpt to be (a massive pretrained markov chain) as all that learning is happening after training
heck, you can make chatgpt pretend to be something that isn't, exhibiting creativity that is well beyond previous llm.
> All it knows, all it knows are the statistical relationships of how words appear together
again, you can ask nonsense, and it will not spew nonsense back. it has a cursory understanding of what nonsense is, and will not fall for easy traps (i.e. what year apollo 7 landed on the moon) that would trick straighforward probabilistic models
all in all, this article toned down my respect for the blog down two pegs, if he's so wrong about this topic, how much was he wrong on other topics where I had no understanding and took his word for good?
This sounds very arrogant. What is your experience grading college essays? (In other words, you are talking about completely differet use-cases)
If we imagine a universe with only one thing (difficult because we would be there too, but bear with me), what could we know about that one thing? It would be the whole universe, all and nothing at the same time.
I would like to propose an alternative take on the same: we also are incapable of knowing. We are just putting in relationship one thing with others, in a similar way as ChatGPT, but on a much bigger scale. And that is fine.
We are still special: we have sensors (senses), we can move, feel pain, but from the point of view of knowledge we are still comparing things and putting them in relationship to one another.
No value judgement on this specific case, but this is commonly known as Gell-Mann amnesia: http://www.wikibin.org/articles/gell-mann-amnesia-effect.htm...
More seriously, I have so far no reason to believe that he's wrong about neither him nor his students having managed to make ChatGPT produce an even passable essay, so if you want to prove him wrong...
if we can't understand that, I have a deep horror for the atrocities we may commit if/when we do create machine subjectivity.
There's a philosophical concept, "zimboes", which is useful here - philosophical zombies which believe that they are conscious, and believe that they experience subjective understanding. I don't follow all the arguments, but from what I understand, they're used to argue for "if it quacks like a duck" presentations of consciousness, rather than some more abstract criteria.
This begs the question (if you made it to the footnotes of the article discussed here, you will know what I mean).
More precisely, you start your argument by begging me to take for granted that it can emulate conscious beings - no, you will have to proof that, and not by making it into an axiom.
It's a simple extension from "it mimics human text" to "what if it got so much better we can't tell the difference". If we can't tell the difference, definitionally it may as well be conscious. You can disagree with the axiom that it does mimic human text, but generating text with it looks like a pretty good mimicry to me.
You may not agree that it will ever get that good - I would also agree with that. I don't think the proposed scenario is at all likely, certainly not in my lifetime, but if it was, I'd be there lobbying for the large language models to have rights.
Re: ChatGPT, what can it offer in entertainment for house pets?
Teach, then?
I don't know this professor, but many college professors themselves could be replaces with ChatGPT.
Students will find ways to make their lives "easier" by sabotaging their own education with assistants like AI, regardless of the quality of professors teaching them.
Tools like ChatGPT can only regurgitate what was already written, often with confident but wrong takes. They can't offer new perspectives, or inspire students with a different way of thinking about the world. These are not teacher, let alone college professor, replacements.
Yeah, just like many professors
i think this remains to be seen.
At least, if you asked chatGPT for recipes, they can produce recipes that aren't pre-existing. So for at least some small domain, it can offer new perspective. Whether those recipes tastes good is a different story...
That is not how ChatGPT works. It doesn't just regurgitate data it has seen. It its capable of offering new perspectives. Just ask it.