Richard Stallman on ChatGPT
stallman.org
stallman.org
I'm sure while the AI lab folks didn't have the techniques and compute to do ML like we do now, they have thought a lot about the definition of intelligence and what it takes to achieve it, going from the narrow task-specific AIs to truly generic intelligence.
ML/RL allowed us to create systems that given a train/test set, can learn underlying patterns and discover connections in data without a developer having to program it.
Transformers/LLMs are a scaled up version of the approach (I don't want to get into the weeds as it's beside the point)
Stallman asserts LLMs fall short of general intelligence, which I think he has a much better understanding of what that entails than most people give him credit for. Considering his AI past, I'd be surprised if he didn't keep up with the advancements and techniques in the field at least to some degree (or that understanding said techniques would be beyond his ability).
Because of this, I'm running with the theory that he knows what he's talking about even he doesn't elaborate on it here.
In particular the quip that it's really just a "bullshit generator" is 100% correct. But also true for, y'know, all the intelligent humans on the planet.
At the end of the day AI gets stuff wrong, as far as we can tell, for basically the same reasons that we get stuff wrong. We both infer from intuition to make our statements about life, and bolt "reasoning" and "logic" on as after the fact optimizations that need to be trained as skills.
(I'm a lot more sympathetic to the free software angle, btw. The fact that all these models live and grow only within extremely-well-funded private enclosures is for sure going to have some very bad externalities as the technology matures.)
That's not true.
> At the end of the day AI gets stuff wrong, as far as we can tell, for basically the same reasons that we get stuff wrong.
Also not true.
> We both infer from intuition [...]
Also not true.
If you need more information, you can search them or ask an LLM.
Here you go:
https://chatgpt.com/share/69384dce-c688-8000-97f7-23d0628cd5...
He's not comparing them to humans, he attributes knowledge/understanding (enough to pass his bar for "is AI") to yolov5 models, xgboost trained trees and as far as I can tell closed source transformer based models too. But not ChatGPT.
And it's certainly not a boon for freedom and openness.
Don't get me wrong I think they are remarkable but I still prefer to call it LLM rather than AI.
A dog's cognitive capabilities are nowhere near human level. Is a dog intelligent?
An LLM is a very clever implementation of autocomplete. The truly vast amount of information we've fed it provides a wealth of material to search against, the language abstraction allows for autocompleting at a semantic level and we've add enough randomness to allow some variation in responses, but it is still autocomplete.
Anyone who has used an LLM enough in an uncommon domain they are very familiar with has no doubt seen evidence of the machine behind the curtain from faulty "reasoning" where it sometimes just plays madlibs to a complete lack of actual creativity.
> I call it a "bullshit generator" because it generates output "with indifference to the truth".
And if we follow the link we find he's referring to LLMs:
> “Bullshit generators” is a suitable term for large language models (“LLMs”) such as ChatGPT, that generate smooth-sounding verbiage that appears to assert things about the world, without understanding that verbiage semantically. This conclusion has received support from the paper titled ChatGPT is bullshit by Hicks et al. (2024).
No one thinks the database, orchestration, tool, etc. portions of ChatGPT are intelligent and frankly, I don't think anyone is confused by using LLM as shorthand not just for the trained model, but also all the support tools around it.
Acting Intelligent, works for me.
For one, it needs multiple orders of magnitude more tokens (millions? billions? more?) to start following grammar rules: watch any kid pick it up with exposure to maybe 10,000 words by the age of 2 (look how they apply the rule to exceptions and while it does not work, it shows their internalization of the rule).
Next, "how" LLMs work is evident when it barfs in gotcha questions like "if yesterday was Tuesday, what day was yesterday" (I believe all major players have overcome this one in the last 12 months, but you can always find these LLM-twisters: yes, you can trick humans too by distracting them, but these are computers which "don't pay attention"?)
And it can frequently undo an improvement it added earlier, or mess something new up that you did not call out specifically.
Any test you can device for this, ChatGPT would reliably pass if the medium was text, while a good fraction of humans might actually fail. It does a pretty good job if the medium was audio.
Video, and in person remains slightly out of reach for now. But I doubt we are not going to get there eventually.
That's clearly untrue unless you qualify "test" as "objective automated test." Otherwise, "convince Stallman you have intelligence according to his definition," is a test that ChatGPT hasn't passed and which every human probably would:
> I define "intelligence" as being capable of knowing or understanding, at least within some domain. ChatGPT cannot know or understand anything, so it is not intelligence. It does not know what its output means. It has no idea that words can mean anything.
I understand the sentiment, but reality is that it does with words pretty much what you’d expect a person to do. It lacks some fundamentals like creativity and that’s why it’s not doing real problem solving tasks, but it’s well capable of doing mundane tasks that the average person gets paid to do.
And when it comes to trust and accuracy, if I ask it a question about German tax system, it will look up sources and may give an answer with an inaccuracy or two but it will for sure be more accurate than whatever I will be able to do after two hours of research.
I don't think that's an appropriate analogy at all.
He's not saying that AI is not useful. He's saying that it doesn't understand or reason, so it does not generate the truth.
So you can ask chatgpt a question about the german tax system, but it would be a mistake to have it do your taxes.
in the same way, a calculator could help with your taxes, because it has been engineered to give precise answers about some math operations, but it cannot do your taxes.
It’s equally true for humans, benchmarks of intelligence. Most shortcomings in our working life is from miscommunications and misunderstanding requirements, and then simply by incompetence and making trivial mistakes.
At the same time, LLMs are not a bullshit generator. They do not know the meaning of what they generate but the output is important to us. It is like saying a cooker knows the egg is being boiled. I care about the egg, cooker can do its job without knowing what an egg is. Still very valuable.
Totally agree with the platform approach. More models should be available to be run own own hardware. At least 3rd party cloud provider hardware. But Chinese models have dominated this now.
ChatGPT may not last long unless they figure out something, given the "code red" situation is already in their company.
Except that LLMs have no mechanism for transparent reasoning and also have no idea about what they don't know and will go to great lengths to generate fake citations to convince you that it is correct.
Humans have transparent reasoning?
> and also have no idea about what they don't know
So why can they respond saying they don't know things?
Because sometimes, the tokens for "I don't know" are the most likely, given the prior context + the RLHF. LLMs can absolutely respond that they don't know something or that they were incorrect about sometimes, but I've only seen that happen after first pointing out that they're wrong, which changes the context window to one where such an admission of fault becomes probable.
For things I don't understand deeply, I can only look if it sounds plausible and realistic, but I can't have full trust.
The "language" it uses when it's wrong is still just an extension of the token-completion it does (because that's what text contains in many of the online discussions etc).
Isn't that a good definition of what bullshit is?
They are a bullshit generator. And "the output" is only important for the CIA.
I have much respect for him but this is at the level of old-man-shouting-at. Criticism should be more targeted and not just rehashing the same arguments, even if true.
> Doing your own computing via software running on someone else's server inherently trashes your computing freedom.
As always with Stallman he is dogmatic well past the point of reasonableness.
I'm not sure if these models are trained using unsupervised learning and are capable of training themselves to some degree, but even if so, the learning process of gradient descent is very inefficient, so by the commonly understood definition of intelligence (the ability to figure out and unfamiliar situation), the intelligence of an inference only model is zero. Models that do test time training might be intelligent to some degree, but I wager their current intelligence is marginal at best.
But also he does count much simpler systems as AI, so it's not about learning on the fly or being anything like human intelligence.
Yeah. Some people are intelligent, but never learn. /s
It's a a qualitatively better intelligence.
An intelligence that can solve problems that fall into its training set better is quantitatively better.
Likewise, an intelligence that learns faster is quantitatively better.
To give a concrete and simple example, take a simple network trained to recognized digits. The network can be of arbitrary quality, it can be robust or not, fast or slow, but it can't do more than digits.
Another NN that can learn to recognize more symbols is a more general kind of AI, which again introduces another set of qualitative measures, namely how much training does it need to learn a new symbol robustly.
'Intelligence' is a somewhat vague term as any of the previous measures I've defined could be called intelligence (training accuracy, learning speed, inference speed etc., coverage of training set).
You could claim a narrower kind of intelligence that exists without learning (which is what ChatGPT is and what you gave as example with the person that only has a short-term memory) is still intelligence, but then we are arguing semantics.
Inference only LLMs are clearly missing something and are lacking in generality.
> To give a concrete and simple example, take a simple network trained to recognized digits. The network can be of arbitrary quality, it can be robust or not, fast or slow, but it can't do more than digits.
This is the kind of thing he would class as AI, but not LLMs.
This argument does a great job anthropomorphizing ChatGPT while trying to discredit it.
The part of this rant I agree with is "Doing your own computing via software running on someone else's server inherently trashes your computing freedom."
It's sad that these AI advancements are being largely made on software you can not easily run or develop on your own.
For many people and purposes, it does indeed serve the same purpose. I use it all the time for coding, which is still very tricky, and for writing emails. For writing emails in particular, it is already a life-changing technology. I have always wanted my own secretary to dictate and finalize various letters. But, for some reason, companies don't provide secretaries anymore. Now, I can finally have an LLM instead. I guess there's no discussion that a good secretary must have always been quite intelligent.
Does only direct sensory input count, or does experience mediated through tools count? How much do you or I really know about the Oort Cloud?
A lot of people mysticise them or want to quibble about intelligence, and then we end up out in the "no true Scottsman" weeds.
We know what they are, how they work, what they generate. Let them be their own thing, do not anthropomorphize them.
The thing we don't understand so well is humans, really.
We are mostly autocomplete with a mild capacity to synthesise new ideas. It’s the network effect of communicating that creates the feedback loops which amplify our collective intelligence.
Also if you think intelligent life has to have a regard for truth, not just a regard for self, tune in to any news channel.
This reads like more a petulant rant than a cogent and insightful analysis of those issues.
For those who will take “bullshit” as an argument of taste I strongly suggest taking a look at the referenced work and ultimately Frankfurt’s, to see that this is actually a pretty technical one. It is not merely the systems’ own disregard to truth but also its making the user care about the truthiness less, in the name of rhetoric and information ergonomics. It is akin to the sophists, except in this case chatbots couldn’t be non-sophists even they “wanted” to because they can only mimic relevance, and the political goal they seem to “care” about is merely making other use them more - for the time being.
Computing freedom argument likewise feels deceptively about taste but I believe harsh material consequences are yet to be experienced widely. For example I was experiencing a regression I can swear to be deliberate on gemini-3 coding capabilities after an initial launch boost, but I realized if someone went “citation needed” there is absolutely no way for me to prove this. It is not even a matter of having versioning information or output non-determinism, it could even degrade its own performance deterministically based on input - benchmark tests vs a tech reporter’s account vs its own slop from a week past from a nobody-like-me’s account - there is absolutely no way for me to know it nor make it known. It is a right I waived away the moment I clicked “AI can be wrong” TOS. Regardless of how much money I invest I can’t even buy a guarantee on the degree of average aggregate wrongness it will keep performing at, or even knowledge thereof, while being fully accountable for the consequences. Regression to depending on closed-everything mainframes is not a computing model I want to be in yet cannot seem to escape due to competitive or organizational pressures.
Can you describe what you mean by this more? Like you think there was some kind of canned override put in to add a regression to its response to whatever your input was? genuine question
User has two knobs called the thinking level and the model. So we know there are definitely per call knobs. Who can tell if thinking-high actually has a server side fork into eg thinking-high-sports-mode versus thinking-high-eco-mode for example. Or if there were two slightly different instantiations of pro models, one with cheaper inference due to whatever hyperparameter versus full on expensive inference. There are infinite ways to implement this. Zero ways to be proven by the end user.
Saying "it's not intelligence" it's the wrong framing.
also, there are fully open LLM, including the training data.
He was right on a number of things, very important ones, but he's loosing it with old age, as we all will.
What you're talking about is "The Singularity", where a computer is so powerful it can self-advance unassisted until the entire planet is paperclips. There is no one claiming that ChatGPT has reached or surpassed that point.
Human-like intelligence is a much lower bar. It's easy to find arguments that ChatGPT doesn't show it (mainly it being incapable of learning actively, and with there being many ways to show it doesn't really understand what it's saying either), but a Human cannot create ChatGPT 2.0 on request, so it follows to reason a human-like intelligence doesn't necessarily have to be able to do so either.
> There are systems which use machine learning to recognize specific important patterns in data. Their output can reflect real knowledge (even if not with perfect accuracy)—for instance, whether an image of tissue from an organism shows a certain medical condition, whether an insect is a bee-eating Asian hornet, whether a toddler may be at risk of becoming autistic, or how well a certain art work matches some artist's style and habits. Scientists validate the system by comparing its judgment against experimental tests. That justifies referring to these systems as “artificial intelligence.”
This is nowhere near arguing that it should be able to make new versions of itself.
Mundane for Dec 2025.
Its a mistake to expect too much from it now though or treat it as some sort of financial cost-cutting panacea. And its a mistake being played right now by millions, spending trillions that may end up in financial crash when reality checks back that will make 2008 crisis look like a children's game.
In the labs they’ve surely just turned them on full time to see what would happen. It must have looked like intelligence when it was allowed to run unbounded.
Separate the product from the technology and the tech starts to get a lot closer to looking intelligent.
The definitions of "knowing" and "understanding" are being challenged.
Also, it's no longer possible to not have a dependency on other opaque softwares.
> The liar, Frankfurt holds, knows and cares about the truth, but deliberately sets out to mislead instead of telling the truth. The "bullshitter", on the other hand, does not care about the truth and is only seeking "to manipulate the opinions and the attitudes of those to whom they speak"[0]
[0] https://en.wikipedia.org/wiki/Bullshit#Harry_Frankfurt's_con...
They also can be massaged for financial incentives when controlled by private corporations, which can result in prioritizing things other than truth, much like humans.
This is what RMS is flagging, though not very substantiated.
And you can run some models locally. What does he think of open-weight models - there is no source code to be published. Closest thing - the training data - needs so many resources to turn into weights that it's next to useless.
I would say his main argument is that you should not use closed source software.
I am not a cash poor college student, I share his philosophy, hence why I try to use Free Software when I have the choice.
Unlike Stallman, I am not a zealot.
His argument/s go beyond that and {wa,i}s actively hostile towards uses and interoperability if code, other libraries, systems, services, and users aren't pure enough. That was the progression from GPL2 -(TiVo)-> GPL3 -(Mandriva )-> AGPL with ever more extremism and less choice and usability.
There's no need for anyone to mirror his entire philosophy or much of it, but that some bits can be replaced by the choice of the user/developer without being forced by others into another walled garden (licensing tainting) that claims to be "free".
YMMV. Take with a grain of salt. Ask your doctor if licensing is right for you.
:peace:
I'm personally a fan of a DWFL license model of FOSS as a gift and giving up control rather than an exclusionary battle for utopian purity. Either create secret groups of code sharing that aren't public or give code away; trying to tell other people what they can do with gifts is as absurd as subscriptions for windshield wipers. I guess I'm neither a communist nor Marxist, sadly, merely a variation of a socialist who values freedom and choice not imposed by others. At the rate the current crop of billionaires are mismanaging things and running roughshod over us little people, the blowback in some land(s) will lead to a "Soviet Union Dve" rhyming of history with even more strife, same as the old strife.
:peace-emoji-here:
Whats bad about: RMS Not making a decent argument make your position look unserious
The objection that is generally made to RMS is that he is 'radically' pro-freedom rather than be willing to compromise to get 'better results'. This is something that makes sense, and that he is a beacon for. It seems such argument weaken even this perspective.
> ChatGPT is not "intelligence", so please don't call it "AI".
Totally ignoring the history of the field.
> ChatGPT cannot know or understand anything
Ignoring large and varied debates as to what these words mean.
From the link about bullshit generators
> There are systems which use machine learning to recognize specific important patterns in data. Their output can reflect real knowledge (even if not with perfect accuracy)—for instance, whether an image of tissue from an organism shows a certain medical condition, whether an insect is a bee-eating Asian hornet, whether a toddler may be at risk of becoming autistic, or how well a certain art work matches some artist's style and habits. Scientists validate the system by comparing its judgment against experimental tests. That justifies referring to these systems as “artificial intelligence.”
Feels absurd to say LLMs don't learn patterns in data and that the output of them hasn't been compared experimentally.
We've seen this take a thousand times and it doesn't get more interesting to hear it again.
What does that mean? "Others have called such tools AI" is argumentum ad populum and a fallacious argument.
> Ignoring large and varied debates as to what these words mean.
Lacking evidence of ChatGPT knowing or understanding things, that is the null hypothesis.
He's famously a curmudgeon, not lazy. How would you expect him to respond?
> Totally ignoring the history of the field.
This criticism is so vague it becomes meaningless. No-one can respond to it because we don't know what you're citing exactly, but you're obviously right that the field is broad, older than most realise, and well-developed philosophically.
> Ignoring large and varied debates as to what these words mean.
Stallman's wider point (and I think it's safe to say this, considering it's one that he's been making for 40+ years) would be that debating the epistemology of closed-source flagship models is fruitless because... they're closed source.
Whether or not he's correct on the epistemology of LLMs is another discussion. I agree with him. They're language models, explicitly, and embracing them without skepticism in your work is more or less a form of gambling. Their undeniable usefulness in some scenarios is more an indictment of the drudgery and simplicity of many people's work in a service economy than conclusive evidence of 'reasoning' ability. We are the only categorically self-aware & sapient intelligence, insofar as we can prove that we think and reason (and I don't think I need to cite this).
Not lazily, clearly. You can argue he's not lazy, but this is a very lazy take about LLMs.
> Stallman's wider point (and I think it's safe to say this, considering it's one that he's been making for 40+ years) would be that debating the epistemology of closed-source flagship models is fruitless because... they're closed source.
You are making that point for him. He is not. He is actively making this fruitless argument.
> This criticism is so vague it becomes meaningless. No-one can respond to it because we don't know what you're citing exactly, but you're obviously right that the field is broad, older than most realise, and well-developed philosophically.
I don't get what you are missing here then. It's a broad field and LLMs clearly are within it, you can only say they aren't if you don't know the history of the field which is either laziness or deliberate in this case because RMS has worked in the field. I notice he conveniently puts some of his kind of work in this field as "artificial intelligence" that somehow have understanding and knowledge.
> embracing them without skepticism in your work
That's not a point I'm arguing with.
> as we can prove that we think and reason (and I don't think I need to cite this).
Can we? In a way we can test another thing? This is entirely distinct from everything else he's saying here as the threshold for him is not "can think and reason like a person" but the barest version of knowledge or understanding which he attributes to exceptionally simpler systems.
Feel free to check out a longer analysis [1] (which he also linked in the source).
> You are making that point for him. He is not. He is actively making this fruitless argument.
Are we reading the same thing? He wrote:
> Another reason to reject ChatGPT in particular is that users cannot get a copy of it. It is unreleased software -- users cannot get even an executable to run, let alone the source code. The only way to use it is by talking to a server which keeps users at arm's length.
...and you see no connection to his ethos? An opaque nondeterministic model, trained on closed data, now being prepped (at the very least) to serve search ads [2] to users? I can't believe I need to state this, but he's the creator of the GNU license. Use your brain.
> I don't get what you are missing here then. [...] I notice he conveniently puts some of his kind of work in this field as "artificial intelligence" that somehow have understanding and knowledge.
You're not making an argument. How, directly and in plain language, is his opinion incorrect?
> Can we? In a way we can test another thing [...] to exceptionally simpler systems.
Yes... it is one of very few foundational principles and the closest thing to a universally agreed idea. Are you actually trying to challenge 'cogito ergo sum'?
[1] https://www.gnu.org/philosophy/words-to-avoid.html#Artificia... [2] https://x.com/btibor91/status/1994714152636690834
You seem very confused about what I'm saying so I will try again, despite your insult.
It is extremely clear why he would be against a closed source thing regardless of what it is. That is not in any sort of a doubt.
He however is arguing about whether it knows and understands things.
When you said "debating the epistemology of closed-source flagship models is fruitless" I understood you to be talking about this, not whether closed source things are good or not. Otherwise what did you mean by epistemology?
> Feel free to check out a longer analysis [1] (which he also linked in the source).
Yes, I quoted it to you already.
> You're not making an argument. How, directly and in plain language, is his opinion incorrect?
They are AI systems by long standing use of the term within the field.
> Yes...
So we have a test for it?
> it is one of very few foundational principles and the closest thing to a universally agreed idea. Are you actually trying to challenge 'cogito ergo sum'?
That is not a test.
I'm also not sure why you included the words "to exceptionally simpler systems" after snipping out another part, that doesn't make a sentence that works at all and doesn't represent what I said there.
I'd call it an observation, but I'm willing to add that you are exhausting. Confusion (or, more likely a vested interest) certainly reigns.
> It is extremely clear [...] Otherwise what did you mean by epistemology?
We are talking about both because he makes both points. A) Stallman states it possesses inherently unreliable knowledge and judgment (hence gambling) and B) When someone is being imperious there is a need to state the obvious to clarify their point. You understood correctly and seem more concerned with quibbling than discussion. Much in the same way as your persnickety condescension I now wonder if you know and understand things in real terms or are simply more motivated by dunking on Stallman for some obscure reason.
> They are AI systems by long standing use of the term within the field.
No. ChatGPT is not. It is marketed (being the operative term) as a wide solution; yet is not one in the same manner as the purposeful gearing (whatever the technique) of an LLM towards a specific and defined task. Now we reach the wall of discussing a closed-source LLM, which was my point. What I said previously does not elide their abstract usefulness and obvious flaws. Clearly you're someone familiar, so none of this should be controversial unless you're pawing at a discussion of the importance of free will.
> Yes, I quoted it to you already.
I'm aware. Your point?
> That is not a test.
The world wonders. Is this some sort of divine test of patience? Please provide an objective rubric for the proving the existence of the mind. Until then, I'll stick with Descartes.
> I'm also not sure why you included the words "to exceptionally simpler systems" after snipping out another part, that doesn't make a sentence that works at all and doesn't represent what I said there.
Must I really explain the purpose of an ellipsis to you? We both know what you said.
He does. I am talking about one of those points. The epistemology side is entirely unrelated to whether closed source things are good or bad.
> A) Stallman states it possesses inherently unreliable knowledge and judgment (hence gambling)
He says it does not contain knowlegde, not that it is simply unreliable. He happily says unreliable systems have knowledge.
> No. ChatGPT is not. It is marketed (being the operative term) as a wide solution;
A wide solution is nothing to do with things being called AI. This is true both in the field of AI and how RMS defines AI. You can argue that it doesn't meet some other bar that you have, I don't really care, I was responding to his line and why it does not make sense.
> The world wonders. Is this some sort of divine test of patience? Please provide an objective rubric for the proving the existence of the mind. Until then, I'll stick with Descartes.
You said we could prove we could think. I asked if we can do that *in a way we can test other things to see if they can think* because I do not think such a thing exists. You said yes, we do have a test. Now you're complaining that such a thing does not exist
> Now we reach the wall of discussing a closed-source LLM, which was my point.
Ok, it's nothing to do with what I'm talking about though (which should have been extremely clear from what I wrote originally, and given my explicit clarifications to you), so you can have that conversation with someone else.
> Must I really explain the purpose of an ellipsis to you? We both know what you said.
You need to explain how you're using them, because what you're doing to my sentences makes no sense. Here's what I said, and why, broken down for you:
> Can we? In a way we can test another thing? This is entirely distinct from everything else he's saying here
The question of thinking is entirely distinct from what he is saying. It is not at all related, it is a side thing I am responding to you about.
> as the threshold for him is not "can think and reason like a person"
This is key. We are not talking about comparisons to people. He does not talk about thinking or reasoning. He is talking about knowledge and understanding.
> but the barest version of knowledge or understanding
And not even a large amount, we are not comparing to humans, smart humans, animals, etc.
> which he attributes to exceptionally simpler systems.
He says trained yolov5 models have knowledge and are AI.
He says that xgboost models have it and are AI.
He says that transformer based and closed source models are AI.
I've been saying the same thing for years, but it's utterly hopeless by now. Even the critics use that ludicrous "AI" moniker.
Sure we humans don't do this... right?
Seems unnecessary harsh. ChatGPT is a useful tool even if limited.
GNU grep also generates output ”with indifference to the truth”. Should I call grep a “bullshit generator” too?
GNU grep respects user arguments and input files to the dot. It is not probabilistic.
An LLM operates a probabilistic process, and provides output which is statistically aligned with a model. Given an input sufficiently different from the training samples, the output is going to be wildly off of any intended result. There is no algorithm.
To have an algorithm, you need to have a concrete way to show than an output is the correct or optimal one.
An LLM is satisfied by providing any random output that passes some subjective "this-does-not-seem-to-be-a-hallucination" test.
Old man yells at cloud _computing_
Great -- another "submarines can't swim" person. [EDIT2: Apparently this is not his position, although it's only clear in a different page he links to. See below.]
By this definition nothing is AI. Quite an ignorant stance for someone who used to work at an AI laboratory.
ETA:
> Please join me in spreading the word that people should not trust systems that mindlessly play with words to be correct in what those words mean.
Please join me in spreading the counterargument to this: The best way to predict a physical system is to have an accurate model of a physical system; the best way to predict what a human would write next is to have a model of the human mind.
"They work by predicting the next word" does not prove that they are not thinking.
EDIT2, con't: So, he clarifies his stance elsewhere [1]. His position appears to be:
1. Systems -- including both "classical AI" systems like chess and machine learning / deep learning systems -- can be said to have semantic understanding, even if they are not 100% correct, if there has been some effort to "validate" the output: to correlate it to reality.
2. ChatGPT and other LLMs have had no effort to validate their output
3. Therefore, ChatGPT and other LLMs have no semantic understanding.
#2 is not stated so explicitly. However, he actually goes into quite a bit of detail to emphasize the validation part in #1, going so far as to describe completely inaccurate systems still count as "attempted artificial intelligence" because they "purport to understand". So the only way #3 makes any sense is for #2 to be presented as stated.
And, #2 is simply and clearly false. All the AI labs go to great lengths to increase the correlation between the output of their AI and the truth ("reduce hallucination"); and have been making steady progress.
So to state it forwards:
1. According to [1], a system's output can reflect "real knowledge" and a "semantic understanding" -- and thus qualify as "AI" -- if someone "validate[s] the system by comparing its judgment against [ground truth]".
2. ChatGPT, Claude, and others have had significant effort put into them to validate them against ground truth.
3. So, ChatGPT has semantic understanding, and is thus AI.
[1] https://www.gnu.org/philosophy/words-to-avoid.html#Artificia...
There at least is not a large contingent of people going around trying to say there is no such thing as swimming beyond what submarines can do...
Anyone calls anything "AI" and I think it is fair to accept that other people trace the line somewhere else.
It's a pointless naming exercise, no better than me arguing that I'm going to stop calling it quicksort because sometimes it's not quick.
It's widely called this, it's exactly in line with how the field would use it. You can have your own definitions, it just makes talking to other people harder because you're refusing to accept what certain words mean to others - perhaps a fun problem given the overall complaint about LLMs not understanding the meaning of words.
By that definition, SQL query planners, compiler optimizers, Google Maps routing algorithms, chess playing algorithms, and so on were all "AI". (In fact, I'm pretty sure SQLite's website refers to their query planner as an "AI" somewhere; by classical definitions this is correct.)
But does an SQL query planner "understand" databases? Does Stockfish "understand" chess? Does Google Maps "understand" roads? I doubt even most AI proponents would say "yes". The computer does the searching and evaluation, but the models and evaluation functions are developed by humans, and stripped down to their bare essentials.
> There are systems which use machine learning to recognize specific important patterns in data. Their output can reflect real knowledge (even if not with perfect accuracy)—for instance, whether an image of tissue from an organism shows a certain medical condition, whether an insect is a bee-eating Asian hornet, whether a toddler may be at risk of becoming autistic, or how well a certain art work matches some artist's style and habits. Scientists validate the system by comparing its judgment against experimental tests. That justifies referring to these systems as “artificial intelligence.”
I've updated my comment in response to this. Basically: It seems his key test is "Is someone validating the output, trying to steer it towards ground truth?" And since the answer re ChatGPT and Claude is clearly "yes", then ChatGPT clearly does count as an AI with semantic understanding, by his definition.
But that definition a machine that understands the words it produces is AI
Imagine you use an ARIMA model to forecast demand for your business or the economy or whatever. It's easy to say it doesn't have a 'world model' in the sense that it doesn't predict things that are obvious only if you understand what the variables _mean_ implicitly. But in what way is it different from an LLM?
I think Stallman is in the same camp as Sutton https://www.dwarkesh.com/p/richard-sutton
On topics with "complicated disagreements", an important way of moving forward is to find small things where we can move forward.
There are a large number of otherwise intelligent people who think that "LLMs work by predicting the next word; therefore LLMs cannot think" is a valid proposition; and since the antecedent is undoubtedly true, they think the consequent is undoubtedly true, and therefore they do not need to consider any more arguments or evidence.
If I can do one thing, it would be to show people that this proposition is not true: a system which did think would do better at the "predict the next word" task than a system which did not think.
You have to come up with some other way to determine if a system is thinking or not.
"I call it a "bullshit generator" because it generates output "with indifference to the truth"."
yeah, no. the point of post-training with RL is precisely to get the truth on many tasks. Many of the answers in post training are judged on whether they are true or not, not just "RLHF / human preference".
Also it's not like human are perfectly commited to truth always itself and we don't question their overall innate "intelligence" sense.
Stallman just doesn't know what post-training is