Jet planes don't fly by flapping their wings...
When I stumble around trying to explain my thinking though I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't? The only response I have is that when the coin doesn't appear you're going to be putting yourself way further behind the starting line then if you thought from the get go that there's no such thing as magic.
I really like this metaphor, it hits home that just because you sense something is human-like doesn't make it so. We're wired to respond to those sorts of things.
> When I stumble around trying to explain my thinking
I had similar difficulties expressing my thoughts in an organized way. The video below from Richard Sutton (the father of Reinforcement Learning [which LLMs use extensively in training]), helped bring order and firm up some of my thoughts and theories.
https://www.youtube.com/watch?v=21EYKqUsPfg
>I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't?
Not to spoil the video, but it isn't at all functionally equivalent. LLMs are an imperfect and somewhat randomized simulation of what the LLM thinks an average person might say in response to a question. Which is another way of saying, most people will on average get worse answers than if they worked on a problem themselves (but they will get that worse answer from the LLM comparatively quickly). Practical people respond to this positioning fairly well, and for others it still lands a bit since no one wants to think they're below average. That said, I'm still working on my positioning a bit as well!
I believe there's a confusion in this thread between what an LLM _does_ and what an LLM _is_. What is does is output a next token. What it is, is a universal function approximator ([1], i.e. a neural net).
With back probation in the neural net, there could be a full state machine being approximated inside the weight. And a state machine is the exact step-wise reasoning the author claims it needs.
[1]: https://en.wikipedia.org/wiki/Universal_approximation_theore...
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
In context: just because our LLMs don't have an explicit 'system 2' component doesnt mean it can't have superhuman reasoning
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
Love to see the forever moving goalposts. One might say they're autonomously moving..
Do we have finally reliable autonomous driving? And available outside the bay area or whatever small place compared to the rest of the world?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".