Reducing the distance of each statistical leap improves “performance” since you would avoid failure modes that are specific to the largest statistical leaps, but it doesn’t change the underlying mechanism. Reasoning models still “hallucinate” spectacularly even with “shorter” gaps.
If I ask you what's 2+2, there's a single answer I consider much more likely than others.
Sometimes, words are likely because they are grounded in ideas and facts they represent.
Yes, and other times they are not. I think the failure modes of a statistical model of a communicative model of thought are unintuitive enough without any added layers of anthropomorphization, so there remains some value in pointing it out.
Put another way, LLMs are good at talking like they are thinking. That can get you pretty far, but it is not reasoning.
It's true that if it's not producing text, there is no thinking involved, but it is absolutely NOT clear that the attention block isn't holding state and modeling something as it works to produce text predictions. In fact, I can't think of a way to define it that would make that untrue... unless you mean that there isn't a system wherein something like attention is updating/computing and the model itself chooses when to make text predictions. That's by design, but what you're arguing doesn't really follow.
Now, whether what the model is thinking about inside that attention block matches up exactly or completely with the text it's producing as generated context is probably at least a little dubious, and its unlikely to be a complete representation regardless.
How so? Transformers are state space models.
Drive my wife crazy as my answer to her questions are always really slow and considered. I have the first think what thought I want to convey, and then think “how do I translate this into words?”
I think a lot of people here think people reason like a mathematical theorem prover, like some sort of platonic ideal rationalist. That’s not how real brains work though.
There are tons of other things you do, like recalling relevant facts related to the new statement and making sure the statement fits into the facts. You go through a timeline, make sure the statement fits into the timeline. You look at the implications of the statement, make sure those fit with other relevant facts. You opt to do these things depending on what the sentence means and implies. This is not just you "noticing" a contradiction, it's a process.
And what does how real brains work mean anyway. You can't compare writers thinking and writing a novel to some six year old writing a paragraph.
That we implement skills, not deficiencies, is a basic concept that is getting to such a level of needed visibility it should probably be inserted in the guidelines.
We implement skills, not deficiencies.
This was the view of Hume (humans as bundles of experience who just collect information and make educated guesses for everything). Unfortunately, it leads to philosophical skepticism, in which you can't ground any knowledge absolutely, as it's all just justified by some knowledge you got from someone else, which also came from someone else, etc., and eventually you can't actually justify any knowledge that isn't directly a result of experience (the concept of "every effect has a cause" is a classic example).
There have been plenty of epistemological responses to this viewpoint, with Kant's view, of humans doing a mix of "gathering context" (using our senses) but also applying universal categorical reasoning to schematize and understand / reason from the objects we sense, being the most well known.
I feel like anyone talking about the epistemology of AI should spend some time reading the basics of all of the thought from the greatest thinkers on the subject in history...
I agree, I think the problem with AI is we don't know or haven't formalized enough what epistemology should AGI systems have. Instead, people are looking for shortcuts, feeding huge amount of data into the models, hoping it will self-organize into something that humans actually want.
This isn’t about epistemology. We are talking about psychology. What does your brain do when we “reason things out”? Not “can we know anything anyway?” Or “what is the correlation between the map and the territory?” Nor anything like that. Just “what is your brain doing when you think you are reasoning?” And “is what an LLM does comparable?
Philosophy doesn’t have answers for questions of applied psychology.
Rigorous language often comes across as pretentious to any layperson, especially when it concerns subjects like philosophy. I don't know what philosophy you've read, but, based on my experience, it's a pretty safe assumption that most AI practitioners do not own a well creased copy of Critique of Pure Reason.
>This isn’t about epistemology. We are talking about psychology. What does your brain do when we “reason things out”?
The only way to compare what our brain does (psychologically or neurologically) to what LLMs or other models do when we "reason things out" is via epistemology, which is to say "how is it possible to reason that out". Asking how our brains do it psychologically or neurologically is really not relevant, as LLMs are not designed the same as our brains.
>Philosophy doesn’t have answers for questions of applied psychology.
I think that expecting philosophy to have any "answers" for topics that include metaphysical questions is unreasonable, yes. But to even bring up "psychology" when discussing generative probability models is unhelpful anthropomorphization.
For me, it feels like I say something, and in saying it, and putting it into words, I have a feeling about whether it is true and supported or not. A qualitative gauge of its correctness. A lot of my reasoning is done this way, trusting that these feelings are based off of a lifetime experience of accumulated facts and the another things currently being considered.
Explain to me how this is different than a neural net outputting a weight for the truthiness of the state space vector?
A good context (any good context) does not necessarily lead to a good output in LLMs. (It does necessarily lead to a better output, but not necessarily a satisfying, proper, decent, consequential one.)
Very certainly not. We ask if the system achieves the goal.
"When we ask if the coprocessor performs floating point arithmetic, we ask if the system achieves the goal (of getting accurate results)". Not, "does the co-processor ask if we have a spare napkin".