Hacker News Reader

Comment by visarga | Hacker News Reader

"The Illusion of Thinking" is bad, but so is the opposite "it's just math and code". They might not be reasoning like humans, but they are not reducible to just code and math either. They do something new, something that just math and code did not do before.

View on HN

I don't really agree. I use multiple LLMs every day and I feel like I get the most mileage out of them when I think about them as exactly that. Super good text transformers that can paraphrase anything that's been posted to the Internet.

There are a complexities beyond that of course. It can compare stuff, it can iterate through multiple steps ("Reason," though when you look at what it's doing, you definitely see how that term is a bit of a stretch). Lots of emergent applications yet to be discovered.

But yeah, it's math (probabilities to generate the next token) and code (go run stuff). The best applications will likely not be the ones that anthropomorphize it, but take advantage of what it really is.

Have you used them to code? I think that is the tipping point where the level of interaction is so high, when the models are troubleshooting. That is when you can start to get a little philosophical.

The models make many mistakes and are not great software architects, but watch one or even two models work together to solve a bug and you'll quickly rethink "text transformers".

All the time. And what stands out is that when I hit a problem in a framework that's new and badly documented, they tend to fall apart. When there's a lot of documentation, StackOverflow discussions etc. for years about how something works they do an excellent job of digging those insights up. So it fits within my mental model of "find stuff on the web and run a text transformer" pretty well. I don't mean to underestimate the capabilities, but I don't think we need philosophy to explain them either.

If we ever develop a LLM that's able to apply symbolic logic to text, for instance to assess an argument's validity, develop an accurate proof step by step, and do this at least as well as many human beings, then I'll concede that we've invented a reasoning machine. Such a machine might very well be a miraculous invention in this age of misinformation, but I know of no work in that direction and I'm not at all convinced it's a natural outgrowth of LLMs which are so bad at math (perhaps they'd be of use in the implementation).

I mean it is like asking a person to white board a framework they have never seen. However when I then say, "go ahead and research the correct methods or recommended methods of doing x with framework y." The models is usually capable of figuring out what to do.

> just code and math either

this is literally what they are

I think it’s the “just” that they are taking issue with. We are “just” neurons. But we demonstrate interesting emergent behaviors that, in principle, can be reduced to firing neurons but in practice we don’t understand and shouldn’t diminish with the word “just”.

fair point!

To insist it's "just code and math" is to fundamentally mistake the rules for the game. Knowing the rules of Go without playing grants no insight into strategy, just as showing someone the code for Conway's Game of Life without execution reveals nothing about the gliders that emerge. The profound dynamics of a system are not found in its static instruction set. To ignore this is a fundamental error, because these are recursive processes that iteratively rewrite their own context, a reality the "it's just X" argument completely fails to grasp. Of course Turing undecidability is the canonical example - can't predict if a program will halt without execution.

What part of an LLM do you think is not made out of code and maths?

I think the point is we wouldnt say "humans dont reason. It is just chemical and eletrical signals"

Right. The error is in thinking that "just code and maths" can't reason.

LLMs very obviously are reducible to "just code and maths". We know that, because that is how they are made.

We call it breakthrough. And it's just math and code. We've had many of those.

Here's something to think about:

Can we consider AI conscious similar to how Hardy recognised Ramanujan's genius? That if AI weren't to be conscious then they won't have the imagination to write what it wrote.

It doesn't take conciousness to predict the next word repeatedly.

That's reductive. Cherry pick the most lucid LLM output and the least lucid human output and I think at this point the former clearly exceeds the latter. If an LLM is "just" predicting the next word repeatedly then what does that say about humans?

Nothing, since human output is adapted (better or worse) to the human's goals in a particular situation, while an LLM produces text that seems like a logical continuation of the text they are given as input. There is a fundamental difference in these two approaches that makes lessons from one very un-applicable from one to the other.

Have you noticed how predictable humans are when they have Transient Global Amensia?

https://www.radiolab.org/podcast/161744-loops/transcript

*The Prisoner of Memory*

In August 2010, Mary Sue Campbell became trapped in the most peculiar prison—a 90-second loop of her own consciousness. After experiencing transient global amnesia, she lost the ability to form new memories, resetting every minute and a half like a human record player stuck on repeat.

What made her condition haunting wasn't just the forgetting, but the eerie predictability. Every cycle, she asked the same questions in identical phrasing: "What's the date?" followed by "My birthday's already passed? Darn." She'd listen to her daughter's explanation, widen her eyes at the mention of paramedics, and declare "This is creepy!" The nurses began mouthing her words along with her.

Dr. Jonathan Vlahos, who treated similar cases, found this mechanical repetition deeply unsettling. It suggested that beneath our illusion of free will lies something more algorithmic—that given identical inputs, the brain produces identical outputs. Mary Sue's loop revealed the uncomfortable truth that our sense of spontaneous choice might be far more fragile than we imagine.

> that given identical inputs

This part doesn’t seem correct. Each 90 seconds she has completely different inputs: sights, sounds, the people she’s around, etc. Seems like it has more to do with her unique condition than any sort of revelation about free will.

Reply on news.ycombinator.com