Paul Graham on LLMs 'Thinking' | Hacker News Reader

Paul Graham on LLMs 'Thinking'

twitter.com

12 pointsfooker30 comments

It’s still not thinking just because we call the reasoning texts “thoughts”. To imply otherwise in the direction of these models truly “thinking” is just a failure to understand the technology.

If you can convince a sufficient number of people of something does it become true?

It does not. But it does make the world that little bit more annoying, like whenever someone posts anything about AI these days there’s that select few that love to pose those absurd “okay but we’re just electrical impulses too, how can we be sure LLMs aren’t actually conscious and alive too?” “”””philosophical”””” questions.

Things like these make my blood boil for how unserious and completely off-the-wall stupid they are.

Hell, someone the other day equated a plane to a bird because both can fly… this is the type of thing we ought to expect from children just learning what philosophy is. Not adults.

But we do say planes 'fly' and not invent another bogus word for it because a plane is not biological :)

In German an airship doesn't "fly", it "fährt". One could have reasonably invented new words for 'flying' depending on whether their motion is caused by flapping wings, buoyancy of light gases, rotating propellers, jet engines, or being fired out of a canon's barrel.

Colloquial German seems to largely use fliegt.

This is the thing about language, you can have strong opinions about it but language evolves through use. Historical attempts to stop this sort of natural drift has just resulted in dead languages nobody speaks like Sanskrit and Latin.

> dead languages nobody speaks like Sanskrit and Latin

There sure are dead languages but these two examples good they are not.

Next they'll tell you they can make life because we're all just an electrochemical soup.

Uh, huh. Sure we are.

Later Wittgenstein said, basically “meaning is use” - words mean their usage, there’s not some root unshiftable core of meaning behind them. If usage shifts because people keep saying LLMs are thinking, then, well they’re thinking.

Yet similarly, Paulg could also come out with “When I put it to sleep, my MacBook really is asleep. We need to better grasp a world where Macs are awake or asleep and perhaps even dreaming”. This points to the issue with assigning the idea of thinking to LLMs, rather than saying they’re calculating or computing or processing when they’re running their digital matrix computations. Something important to us about thinking has nevertheless not been captured in assigning terminology used for the human experience and applying it to an electronic device. And in no world is my thinking process the same thing as an LLM computing an output, even if those outputs lead to the identical sentence “the cat sat on the mat” - functionalists will tell you that this doesn’t matter and functional equivalence is equivalence but it’s hard to believe that without looking at basic empirical facts about what’s behind the outputs.

Ultimately Paul has noticed something about language from his own first principles, not about thinking.

Submarines don’t swim, basically.

I too sometimes wonder what gives these words power.

1) I'm with you in that I'm critical of people who mindlessly say things like "Claude lied to me", and maybe even "ChatGPT hallucinated".

2) On the other hand I'm also critical of people who assert that "Ackshelly you didn't do any 'work' by putting the crate from this table to that table" or "sweet potatoes are not 'potatoes' at all, so don't call them potatoes".

In physics, words like 'work', 'power', 'heat' have very specific meanings. The 'heat death of the universe' is thought to be near absolute zero Kelvin a.k.a. frigging cold, no heat anywhere. The meaning of 'work' in physics has a certain overlap with what is called 'work' outside of physics (to wit, a number of different things). A typical biologist or taxonomist's view on species is much more nuanced than the ordinary guy-with-some-general-educational-background's view on the subject: the latter will tend to see species as an absolute given which science just has to uncover. Sure scientists can err but ultimately the truth of biological species is often conceived as incontrovertible by the informed layman instead of what it is: a tool for thought, a human measure brought in to help make sense of chaotic data. Darwin already said as much.

Maybe physics should never have chosen terms like 'force', 'work', 'power'. As a German speaker let me remark that especially 'power' is a bit puzzling; German has 'Leistung' instead which I think is a bit clearer. But even when scientists go and create their own terms like 'energy' or 'quantum leap', neither of which existed before being introduced with a specific scientific definition attached to them, the public often goes and appropriates them anyway. So "he's got such a cold energy" ("cold" being associated with lack of thermal energy) and "a massive quantum leap" (i.e. "a sizeable advancement" instead of "the smallest possible change") are colloquially possible.

Look at more squishy terms from squishier sciences like 'language' as used by linguists and things become even more difficult. Sure, "the language of Islamic architecture in early medieval Spain" is a nice and valid book title and no linguists will take exception. But talk about the new research on the "language of birds" in the newspaper and you will ruffle some feathers.

One has to acknowledge that people are challenged with finding words to describe a new phenomenon. Sure, we could stop at 'calculate' or 'compute', as in "ChatGPT computed some falsehoods" instead of saying it 'lied'. But even when I say that "ChatGPT's answer was wrong" doesn't 'answer', alone and by itself, already imply a degree of intentionality that is (or should be, as far we can tell) absent from a process that purely relies on solving one big formula with a pinch of randomness thrown in? Then again, "what is the calculator's answer?" was already used in the 1970s when pocket calculators were all the rage.

Poor choices are still poor choices. Words are very curious things with very infinite meanings, shifting between context. There’s probably something to be said about how we shape our languages and how ambiguity of meaning can prevent those who just don’t know from knowing. You know what I mean?

Sure maybe the English language is choosing to adopt the word “thinking” for what is going on here. Some people are choosing “reasoning”. It’s all the same to me though. The thought outputs aren’t any different that the final message for me, regardless of its shape.

I don’t have a better alternative, but I’m pretty sure this just makes people who aren’t in the industry dumber when they see it.

It doesn’t really matter. There’s no way the training loop necessary is sustainable and these things are a fad. How long? Idk. But language will change and there will be a day where using an LLM requires as much training as reading old religious texts. Got to learn more language.

???

Stupid anthropomorphism that only serves to reduce human accountability.

These things behave like humans. Because of their training. And we have no language to discuss their actions.

When my agents refuse to follow my instructions - I ask them why. Typically they will explain that they think my design is misguided and theirs is better. We'll reach a consensus and move forward.

How on earth are we supposed talk about that without using anthropomorphic language?

Lets coin `contexting`

The funny thing about coining words (or anything, even coining coins...) is that it only works if others use it.

So Paul thinks LLMs, think?

We can add you to the list of smart folks who have lost their minds over this topic, and all for a failure to understand the basic computer science on which they're built.

I guess if you get too far away from the metal (and too high up in the boardroom), it's easy to lose touch.

What do you think is going on if not “thinking?” Can you elaborate how the basic computer science that underlies these systems refutes it.

Sure. Study any basic text about how a processor functions. Now watch a dog learn a trick. There is no correspondence.

It's fun sci-fi to think that people are machines, too. Descartes started it.

Basic computer science (or anything really) does not yet provide a definition of thinking.

It's outside its purview, which is the domain of voltage high/low and Boolean logic. Those do not imply, require, or provide thinking.

What is thinking?

Noam Chomsky is a good person to ask. He spent his career asking that.

And we learned that if one is in the business of copying other peoples thoughts, copying thoughts becomes thinking.

Things are moving so fast that I don't think even Paul understands much about the change that is brewing nor it's scale, no one knows what's going to happen.

Lol. He has fully sold out.

This is a very poor take and based on his education and the things he has done - he surely does not think this....he's pumping LLM based startups his/(others) companies are funding...embarassing for him intellectually but probably quite beneficial financially

Look at some of the shit John Carmack now says for God's sake. Throw enough money at anyone and they turn into an idiot

It is unclear to me what a future void of human thinking looks like.

I wish we would stop and ask that question and ponder if that’s what we want.

It feels a lot like power has been concentrating such that a minuscule minority get to set that direction.

>> Look at some of the shit John Carmack now says for God's sake.

Not questioning but can you share a few so that it can serve as warning reference.

What a delusional world we’re heading towards. :(

Or, maybe your understanding is wrong. He may be incentivized to say it, but you haven’t provided any reason to not believe these systems are “thinking.”

Reply on news.ycombinator.com