"In one well-known experiment, a split-brain patient’s left hemisphere was shown a picture of a chicken claw and his right hemisphere was shown a picture of a snow scene. The patient was asked to point to a card that was associated with the picture he just saw. With his left hand (controlled by his right hemisphere) he selected a shovel, which matched the snow scene. With his right hand (controlled by his left hemisphere) he selected a chicken, which matched the chicken claw. Next, the experimenter asked the patient why he selected each item. One would expect the speaking left hemisphere to explain why it chose the chicken but not why it chose the shovel, since the left hemisphere did not have access to information about the snow scene. Instead, the patient’s speaking left hemisphere replied, “Oh, that’s simple. The chicken claw goes with the chicken and you need a shovel to clean out the chicken shed”" [1]. Also [2] has an interesting hypothesis on split-brains: not two agents, but two streams of perception.
[1] 2014, "Divergent hemispheric reasoning strategies: reducing uncertainty versus resolving inconsistency", https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522
[2] 2017, "The Split-Brain phenomenon revisited: A single conscious agent with split perception", https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf
I am morbidly curious how people are going to creatively explain away the more challenging insights AI gives us in to what consciousness is.
Reading/listening to someone like Robert Sapolsky [1] makes me laugh I could have ever hallucinated about such a muddy, not even wrong concept as "free will".
Furthermore, between the brain and, say, the liver there is only a difference of speed/data integrity inasmuch as one cares to look for information processing as basal cognition: neurons firing in the brain, voltage-gated ion channels and gap junctions controlling bioelectrical gradients in the liver, and almost everywhere in the body. Why does only the brain has a "feels like" sensation? The liver may have one as well, but the brain being an autarchic dictator perhaps suppresses the feeling of the liver, it certainly abstracts away the thousands of highly specialized decisions the liver takes each second solving adequately the complex problem space of blood processing. Perhaps Thomas Nagel shouldn't have asked "What Is It Like to Be a Bat?" [2] but what is it like to be a liver.
[1] "Robert Sapolsky: Justice and morality in the absence of free will", https://www.youtube.com/watch?v=nhvAAvwS-UA
[2] https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F
And that’s just the polar opposite of having a meaningful will at all. It is good that you are pretty much deterministic. You shouldn’t be deciding meaningful things randomly. If you made 20 copies of yourself and asked them to support or oppose some essential and important political question (about human rights, or war, or what-have-you) they should all come down on the same side. What kind of a Will would that be that chose randomly?
* in a safe setting with support, of course.
After breaking my arm, split in two, pinching the nerve and making me unable to move it for about a year, I still feel as if the arm is "someone else's", as if I am moving an object in VR, not something which is "me" or "mine".
https://www.mpg.de/research/unconscious-decisions-in-the-bra...
The idea that there should not be any neural activity before a conscious decision is straight-up dualism---the intangible soul makes a decision and neural activity follows it to carry out the decision.
An alternative way of understanding that result is that the neural activity that precedes the "conscious decision" is the brain's mechanism of coming up with that decision. The "conscious mind" is the result of neural activity, right?
if it turns out that true, it’s truly amazing how well we convince ourselves that we’re in control.
but if our brain controls our actions and not our consciousness, then what is the purpose of consciousness?
Perhaps it's a phenomenon that somehow arises independently ex nihilo from sufficiently complex systems, only ever able to observe, unable to act.
Weird to think about.
The short answer is: chemical reactions start a chain reaction of abstraction towards higher and higher forms of collective intelligence.
For some reason, perhaps something with the way the Hilbert space vector obeying the Schrödinger equation which we usually call "the universe" is [2], but also given the ridiculous breadth and depth of possible explorations of the biological pathways [3], "chunks" of matter tend to group together, they group and form stars and planets, but they also group and form formaldehydes and acetaldehydes and many more. Given enough tries, across multiple environments, in some lucky hydrothermal vents abiogenesis was probably started [4]. Once we had the first agential "chunk" of matter, a group of matter which has a definition of the boundary between a "self", no matter how tiny [5], and an exterior environment, it was more of a game of waiting (~4 billion years) for this tiny agent to grow into an agent with higher-order thinking, self-referentiality, metacognition, and the likes.
Neural networks, as in matrix multiplications, are not conscious because they have no mechanism for deciding what is the environment and what is their own, they are a hammer, sitting there, expecting to be used, not a lacrymaria olor [6], exploring the environment for survival and fun. Could we have neural networks in an agent-like architecture starting to behave more like thermostats, setting goals for themselves? Probably.
[1] "From physics to mind - Prof. Michael Levin", https://youtu.be/_QICRPFWDpg?t=85
[2] "Sean Carroll: Extracting the universe from the wave function", https://www.youtube.com/watch?v=HOssfva2IBo
[3] "Nick Lane, 'Transformer : The Deep Chemistry of Life and Death'", https://youtu.be/bEFzUx_j7tA?t=279
[4] https://en.wikipedia.org/wiki/Hydrothermal_vent
Let's assume the premise that a form of neural network is necessary but insufficient to give rise to conscious experience. Then might it not matter whether the medium is physical or digital?
If you answer this with anything other than "we don't yet know", then you'll be wrong, because you'll be asserting a position beyond what science is able to currently provide; all of this is an open question. But hint: the evidence is mounting that yes, the medium might not matter.
Once you take on an information theoretic view of consciousness, a lot of possibilities and avenues of research open up.
https://en.m.wikipedia.org/wiki/Neural_correlates_of_conscio...
I know brain is a neural network. I just don't understand how cold, hard matter can result in this experience of consciousness we are living right now. The experience. Me. You. Perceiving. Right now.
I'm not talking about the relation between the brain and our conscious experience. It's obvious that brain is collecting and computing data every second for us to live this conscious experience. The very experience of perceiving, being conscious? The thing we take for granted the most, for that we're not without it any time, except when we're asleep?
Matter is what it is. A bunch of carbon and hydrogen atoms. How does the experience arise from matter? It can't. It is a bunch of atoms. I know how NNs and biological neurons work, still I don't see any way matter can do that. There must be some sort of non-matter essence, soul or something like that.
Is a bunch of electrochemical charges this thing/experience I am living right now? How can it be? Is Boltzmann brain [1] a sensible idea at all?
The main point is that with the tremendous discoveries of people such as Church/Turing (matter can be organized in such a fashion as to produce computation) [2] and those forgotten from the first page of history, such as Harold Saxton Burr (matter can be animated through bioelectricity), we no longer are bound to a static metaphysics where objects are made from a material which just sits there. It was obviously never the case, but fighting the phantasms of our own speculation is the hardest fight.
Therefore, no, matter is neither cold, nor hard, and we are surely very far from comprehending all the uses and forms of matter. Just look at all the wood objects around you and think how the same material was available also to Aristotle, 2,400 years ago, and to Descartes, 400 years ago, when they were writing their bad metaphysics, yet they were completely unable to think 1% of the wood objects you have readily available nowadays, cardboard and toothpicks included.
And also, yes, you are electrochemical charges, we all are, what else could we be? We looked insanely deep into the brain [4], there is no magic going on. A caveat would be that, yes, probably, we are not running on the first layer, at the level of sodium ions and neurotransmitters, but that the machinery, the brain, gives rise to a simulation: "only a simulation can be conscious, not physical systems" [5].
[1] https://en.wikipedia.org/wiki/Qualia
[2] https://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis
[3] https://en.wikipedia.org/wiki/Harold_Saxton_Burr
[4] "The Insane Engineering of MRI Machines", https://www.youtube.com/watch?v=NlYXqRG7lus
[5] https://www.youtube.com/watch?v=tyVyirT6Cyk https://www.youtube.com/watch?v=SbNqHSjwhfs
> We looked insanely deep into the brain [4], there is no magic going on.
Indeed all computation and input collection and such happen in the brain. I just don't understand how I can experience anything if I'm composed only of matter. How come there happens to be a mind? Indeed the electrochemical charges from visual receptors in the eye will be transmitted and computed and memory and dopamine and all the neurons will fire regardless of whether I'm only matter or not. But how can the experiencing consciousness, 'me' arise from matter?
> only a simulation can be conscious, not physical systems
This is what I'm talking about, only that I don't see why simulations are not physical systems.
> yes, you are electrochemical charges, we all are, what else could we be?
It's nonsensical and unscientific to completely rule out the possibility that we can be something else as well, especially when we can't study it directly, like in the example of soul.
Sure, we have about 2,700 years of tradition speaking of souls (considering the major religions: Christianity, Islam, Buddhism, Hinduism, and Judaism). Where did those 2,700 years got us? Has any religion been able to build a conscious agent starting from basic materials (whatever they consider basic, pixie dust if they will)? Have all this years speaking of souls managed to achieve something meaningful, even as a side effect, that actually improves the quality of life? I'm talking hay [2], indoor plumbing, hook-and-loop fasteners, ibuprofen, GPS, voltmeters, extreme ultraviolet lithography, things that you and I can use and rely on daily. I have read pretty much all the major texts of the major traditions, from Mahābhārata to Summa Theologiae, call it intellectual curiosity. If not for the "bragging rights" to say that I know what filioque or bodhipakkhiyādhammā means, I would regret it, wasted time and pointless eye strain. So no, it's not nonsensical and unscientific to rule out a not even hypothesis such as the "soul" after 2,700+ years without any kind of results and absolute incompatibility with the way we actually interact with the world, scientifically or not: photons, atoms, electromagnetic fields and the like.
[1] https://en.wikipedia.org/wiki/Dissociative_identity_disorder
[2] "The technologies which have had the most profound effects on human life are usually simple. A good example of a simple technology with profound historical consequences is hay.", https://quotepark.com/quotes/1924489-freeman-dyson-like-many...
Even if you accept classic theory (e.g. hemispheric localization and the homunculus) which most experts don't all this suggests is that the brain tries to make sense of the information it has and in sparse environments it fills in.
How does this make our behavior "mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning" as most humans don't have a severed corpus callosum.
The discussion starts with:
"In a healthy human brain, these divergent hemispheric tendencies complement each other and create a balanced and flexible reasoning system. Working in unison, the left and right hemispheres can create inferences that have explanatory power and both internal and external consistency."
The existence of cognitive dissonance suggested in your citation is in no way analogous to "our own explanations about ourselves and our behaviour are mostly lies, fabrications, hallucinations, faulty re-memorization, post hoc reasoning" and in fact supports the opposite.
In the morning when we wake up, we are "booting" up the memories the brain finds and we believe that we have persisted through time, from yesterday to today, yet we are absolutely sure that is a lie: just look at an Alzheimer patient.
We are feeling this self as if it's somewhere above the neck and we feel like this self is looking at the world and sees "out there", yet we are absolutely sure that is a lie: our senses are being overflown by inputs and the brain filters them, shapes a model of the world, and presents that model to the internal model of itself, which gets so immersed into model of the world that starts to believe the model is indeed the world, until the first bistable image [1] breaks the model down.
[1] https://www.researchgate.net/profile/Amanda-Parker-14/public...
But the bottom line is that introspection is not necessarily reliable.
https://www.health.harvard.edu/blog/right-brainleft-brain-ri... :
> But, the evidence discounting the left/right brain concept is accumulating. According to a 2013 study from the University of Utah, brain scans demonstrate that activity is similar on both sides of the brain regardless of one's personality.
> They looked at the brain scans of more than 1,000 young people between the ages of 7 and 29 and divided different areas of the brain into 7,000 regions to determine whether one side of the brain was more active or connected than the other side. No evidence of "sidedness" was found. The authors concluded that the notion of some people being more left-brained or right-brained is more a figure of speech than an anatomically accurate description.
Here's wikipedia on the topic: "Lateralization of brain function" https://en.wikipedia.org/wiki/Lateralization_of_brain_functi...
Furthermore, "Neuropsychoanalysis" https://en.wikipedia.org/wiki/Neuropsychoanalysis
Neuropsychology: https://en.wikipedia.org/wiki/Neuropsychology
Personality psychology > ~Biophysiological: https://en.wikipedia.org/wiki/Personality_psychology
MBTI > Criticism: https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indi...
Connectome: https://en.wikipedia.org/wiki/Connectome
The post you are replying to is talking about the small subset of individuals who have had their corpus callosum surgically severed, which makes it much more difficult for the brain to send messages between hemispheres. These patients exhibit “split brain” behavior that is well studied by experiments and can shed light into human consciousness and rationality.
I think the correct statement is "so far the answer is we don't know"
How or if this generalizes to healthy brains is not super clear, but it does actually provide a good explanatory model for all sorts of self-contradictory behavior (like addiction): the brain has many semi-independent “interests” that are jockeying for overall control of the organism’s behavior. These interests can be fully contradictory to each other.
Correct, ultimately we do not know. But it’s actually a different question than your rephrasing.
Neuroimaging indicates high levels of redundancy and variance in spatiotemporal activation.
Studies of cortices and other tissues have already shown that much of the neural tissue of the brain is general purpose.
Why is executive functioning significantly but not exclusively in the tissue of the forebrain, the frontal lobes?
These offer very different interpretations of cognition and behavior, and the split brain experiments point toward the latter.
> Modularity: [...] The difficulty with this theory is that in typical non-lesioned subjects, locations within the brain anatomy are similar but not completely identical. There is a strong defense for this inherent deficit in our ability to generalize when using functional localizing techniques (fMRI, PET etc.). To account for this problem, the coordinate-based Talairach and Tournoux stereotaxic system is widely used to compare subjects' results to a standard brain using an algorithm. Another solution using coordinates involves comparing brains using sulcal reference points. A slightly newer technique is to use functional landmarks, which combines sulcal and gyral landmarks (the groves and folds of the cortex) and then finding an area well known for its modularity such as the fusiform face area. This landmark area then serves to orient the researcher to the neighboring cortex. [7]
Is there a way to address the brain with space-filling curves around ~loci/landmarks? For brain2brain etc
FWIU, Markham's lab found that the brain is at max 11D in some places; But an electron wave model (in the time domain) may or must be sufficient according to psychoenergetics (Bearden)
> Distributive processing: [...] McIntosh's research suggests that human cognition involves interactions between the brain regions responsible for processes sensory information, such as vision, audition, and other mediating areas like the prefrontal cortex. McIntosh explains that modularity is mainly observed in sensory and motor systems, however, beyond these very receptors, modularity becomes "fuzzier" and you see the cross connections between systems increase.[33] He also illustrates that there is an overlapping of functional characteristics between the sensory and motor systems, where these regions are close to one another. These different neural interactions influence each other, where activity changes in one area influence other connected areas. With this, McIntosh suggest that if you only focus on activity in one area, you may miss the changes in other integrative areas.[33] Neural interactions can be measured using analysis of covariance in neuroimaging [...]
FWIU electrons are most appropriately modeled with Minkowski 4-space in the time-domain; (L^3)t
Neuroplasticity: https://en.wikipedia.org/wiki/Neuroplasticity :
> The adult brain is not entirely "hard-wired" with fixed neuronal circuits. There are many instances of cortical and subcortical rewiring of neuronal circuits in response to training as well as in response to injury.
> There is ample evidence [53] for the active, experience-dependent re-organization of the synaptic networks of the brain involving multiple inter-related structures including the cerebral cortex.[54] The specific details of how this process occurs at the molecular and ultrastructural levels are topics of active neuroscience research. The way experience can influence the synaptic organization of the brain is also the basis for a number of theories of brain function
> Physical observation (via the transverse photon interaction) is the process given by applying the operator ∂/∂t to (L^3)t, yielding an L3 output.
> Recent work has revealed that the neural activity patterns correlated with sensation, cognition, and action often are not stable and instead undergo large scale changes over days and weeks—a phenomenon called representational drift. Here, we highlight recent observations of drift, how drift is unlikely to be explained by experimental confounds, and how the brain can likely compensate for drift to allow stable computation. We propose that drift might have important roles in neural computation to allow continual learning, both for separating and relating memories that occur at distinct times. Finally, we present an outlook on future experimental directions that are needed to further characterize drift and to test emerging theories for drift's role in computation.
So, to run the same [fMRI, NIRS,] stimulus response activation observation/burn-in again weeks or months later with the same subjects is likely necessary given Representational drift.
To begin with, the split-brain experiments don't provide clear or strong evidence for anything given the small sample size, heterogeneity in procedure (i.e. was there complete comissurotomy or just callosotomy) and the elapsed time between neuropsychiatric evaluation and initial procedures which relies on the assumption that adaptation does not occur and neuroplasticity is not a thing. The split-brain experiments are notable because the lab experiments SUGGEST the lack of communication between two hemispheres and a split conscious however this is paradoxical with everyday experience of these patients, far from providing evidence for anything.
Ignoring that for a moment, how do the split-brain experiments support 'the brain has many semi-independent “interests” that are jockeying for overall control of the organism’s behavior'?
How is addiction self-contradictory exactly and what does this have to do with split-brain?
If your point is that different parts of the brain (e.g. the reward system and the executive function regions) have different roles this isn't really debatable, obviously different parts of the brain are all doing their individual jobs and the most widely accepted theory is that these are integrated in some unknown mechanism by a single consciousness which remains in control.
Your original statement of: does the brain know what the brain is doing. The answer so far does not seem to be "yes."
Suggests you're arguing that the brain has many different consciouses that are in a constant battle, i.e. there is not a unified consciousness in control of behavior.
To take your addiction example, the brain is very much aware of what it is doing and addiction is not self-contradictory because short term rewards are being valued above long-term ones and health of the organism. The reward system model provides an excellent neurobiological explanation for addiction.
This is not directly evidenced by either addiction or the split brain experiments and is at best a hypothetical model hence why it's a theory and my original response to your statement.
> The split-brain experiments are notable because the lab experiments SUGGEST the lack of communication between two hemispheres and a split conscious however this is paradoxical with everyday experience of these patients, far from providing evidence for anything.
It is not "paradoxical" but yes it does conflict with some reported experience. However, even healthy individuals often report being "of two minds" or struggling to "make up their [singular] mind." Why are these utterances to be dismissed while the also-subjectively-reported sensation of unitary experience is taken as fact?
> Suggests you're arguing that the brain has many different consciouses that are in a constant battle, i.e. there is not a unified consciousness in control of behavior.
I wouldn't characterize my position as "many different consciousnesses," but rather that consciousness is dispersed across (at least) the brain. In some scenarios (such as a corpus callosotomy) and perhaps in more everyday scenarios - perhaps all day every day - that dispersed activity can fall out of internal "synchronization." Anyway, you provided the exact same interpretation in the previously quoted section: "the lab experiments SUGGEST the lack of communication and a split consciousness."
You just go one step further of prioritizing the subjectively reported sensation of unitary consciousness over also-subjectively-reported sensation of non-unitary consciousness. That's your prior taking hold, not mine, and not actual evidence.
You yourself admit we do not know the mechanism (if any exists) by which the activity in various parts of the brain are integrated. We do not know if this process actually even occurs!
Regarding addiction, it is very, very commonly reported that addicts will go into "autopilot" like states while satisfying their addictions and only "emerge" when they have to face consequences of their behaviors. Again, subjectively reported, but so is the experience of unitary consciousness! If we cannot trust one then we shouldn't take it as granted that we can trust the other.
I get the sense you think you're arguing against some firmly held belief or a model I'm proposing as fact: you're not! We're both saying "we don't know much about how this works." And no, neurochemical mechanisms are not complete answers to how brain activity ladders up to conscious experience, similar to how a molecular model of combustion cannot explain much about urban traffic patterns.
Similarly, the initial comment of 'does the brain know what the brain is doing. The answer so far does not seem to be "yes."' is misleadingly suggesting there is a shred of evidence supporting that the answer is 'no' or that the answer is 'not yes'. There are no answers so far, just questions.
If anything, there are more unified consciousness hypotheses than otherwise, although if you refer back to my original reply I did not make this assertion: 'I think the correct statement is "so far the answer is we don't know"'
> It is not "paradoxical" but yes it does conflict with some reported experience.
Rather than belabour the experiment results and implications here is a great peer-reviewed article by experts in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/
> Regarding addiction, it is very, very commonly reported that addicts will go into "autopilot" like states while satisfying their addictions and only "emerge" when they have to face consequences of their behaviors. Again, subjectively reported, but so is the experience of unitary consciousness! If we cannot trust one then we shouldn't take it as granted that we can trust the other.
The dopamine reward system understanding (which by the way is probably the most well-validated and widely believed model in neurobiology) provides a rational explanation for addiction.
You haven't explained what is self-contradictory, that a few case reports exist of patients claiming they went in and out of consciousness? That's not a contradiction.
From your own linked article: "In short, callosotomy leads to a broad breakdown of functional integration ranging from perception to attention."
If the dopamine reward system is the full answer, then what explains drug addicts anguishing about their addiction while simultaneously actively seeking out their next hit? What part of the brain is producing the anguish if the whole behavior is conclusively described by the dopamine reward system?
Still not sure what the contradiction is, is it regret now? Cause that isn’t contradictory.
It’s pretty simple, drugs feel really good when you take them so a single consciousness prioritizes that feeling over long term interests. When one is not taking them and facing the consequences of those decisions they feel bad. To make the bad feelings go away one takes more drugs and the cycle repeats.
There is no "their" and there is no "thought process" . There is something that produces text that appears to humans like there is something like thought going on (cf the Eliza Effect), but we must be wary of this anthropomorphising language.
There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text.
What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?
- The argument that LLMs are missing introspection / inner voice is based on attempting to compare LLMs directly with human minds.
- Human minds have conscious and unconscious parts; for many people, part of the boundary between unconscious and conscious mind manifests as the "inner voice" - the one that makes verbalized thoughts "appear" in their head (or rather perhaps become consciously observed).
- Based entirely on my own experience in using GPT-3.5 and GPT-4, and my own introspection, I feel that GPT-4 bears a lot of resemblance to my inner voice in terms of functioning.
- Therefore I propose that comparing LLMs directly to human minds is unproductive, and it's much more interesting/useful to compare them to the inner voice in human minds: the part of the boundary between unconscious and conscious that uses natural language for I/O.
And when Hinton says at MIT, "I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time," I believe he's commenting on the ability of LLM's to think on some level.
1. Show GPT-4 a GPT-produced text with the activation level of a specific neuron at the time it was producing that part of the text highlighted. They then ask GPT-4 for an explanation of what the neuron is doing.
Text: "...mathematics is done _properly_, it...if it's done _right_. (Take ..."
GPT produces "words and phrases related to performing actions correctly or properly".
2. Based on the explanation, get GPT to guess how strong the neuron activates on a new text.
"Assuming that the neuron activates on words and phrases related to performing actions correctly or properly. GPT-4 guesses how strongly the neuron responds at each token: '...Boot. When done _correctly_, "Secure...'"
3. Compare those predictions to the actual activations of the neuron on the text to generate a score.
So there is no introspection going on.
They say, "We applied our method to all MLP neurons in GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with explanations that scored at least 0.8, meaning that according to GPT-4 they account for most of the neuron's top-activating behavior." But they also mention, "However, we found that both GPT-4-based and human contractor explanations still score poorly in absolute terms. When looking at neurons, we also found the typical neuron appeared quite polysemantic."
What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads to similar "thoughts" and/or navigation of the concept web, and therefore the thoughts are continuous in a sense.
But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.
When someone states definitively what LLMs can or cannot do, that is when you know to immediately disregard them as the waffling of uninformed laymen lacking the necessary knowledge foundations (cognitive/neuroscience/philosophy) to even appreciate the uncertainty and finer points under discussion (all the open questions regarding human cognition etc).
They don't know what they don't know and make unfounded assertions as result.
Many would do to refrain from speaking so surely about matters they know nothing about, but that is the internet for you.
To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step?
I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be replicated. Not even with 175,000,000,000 calculations for every word generated.
All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.
Above you said about AI LMs:
> There is no "their" and there is no "thought process"
So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special.
> All this talk of AGI and sentience and so on is premature and totally unfounded
I don't see any mention of AGI or sentience in this thread?
Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition:
I think LLMs are "Semantic Clouds of Words" + grammar and syntax generator. Someone could just discard the grammar and syntax generator, just use the semantic cloud and create the grammar and syntax by himself.
For example, in writing a legal document, a slightly educated person on the subject, could just use the relevant words put into an empty paper, fill in the blanks of syntax and grammar, alongside with the human reasoning which is far superior than any machine reasoning, till today at least.
The process of editing the GPT* generated documents to fix reasoning is not a negligible task anyway. Sam Altman mentioned that: "the machine has some kind of reasoning", not a human reasoning ability by any means.
My point is, that LLMs are two programs fused into one, "word clouds" and "syntax and grammar", sprinkled with some kind of poor reasoning. Their word clouding ability, is so unbelievable stronger than any human it fills me with awe every time i use it. Everything else is, just whatever!
Looking at it this way, I honestly wouldn't be surprised if that's exactly how "System 1" (to borrow a term from Kahneman) in our brains works.
What I'm saying is:
> In my opinion, in case there is a way to extract "Semantic Clouds of Words", i.e given a particular topic, navigate semantic clouds word by word, find some close neighbours of that word, jump to a neighbour of that word and so on, then LLMs might not seem that big of a deal.
It may be much more of a deal than we'd naively think - it seems to me that a lot of what we'd consider "thinking" and "reasoning" can be effectively implemented as proximity search in a high-dimensional enough vector space. In that case, such extracted "Semantic Cloud of Words" may turn out to represent the very structure of reasoning as humans do it - structure implicitly encoded in all the text that was used as training data for the LLMs.
Yes, exactly that. That's what GPT4 is doing, over billions of parameters, and many layers stacked on top of one another.
Let me give you one more tangible example. Suppose Stable Diffusion had two steps of generating images with humans in it. One step, is taking as input an SVG file, with some simple lines which describe the human anatomy, with body position, joints, dots as eyes etc. Something very simple xkcd style. From then on, it generates the full human which corresponds to exactly the input SVG.
Instead of SD being a single model, it could be multimodal, and it should work a lot better in that respect. Every image generator suffers from that problem, human anatomy is very difficult to get right.[1] The same way GPT4 could function as well. Being multimodal instead of a single model, with the two steps discreet from one another.
So, in some use cases, we could generate some semantic clouds, and generate syntax and grammar as a second step. And if we don't care that much about perfect syntax and grammar, we feed it to GPT2, which is much cheaper to run, and much faster. When i used the paid service of GPT3, back in 2020, the Ada model, was the worst one, but it was the cheapest and fastest. And it was fast. I mean instantaneous.
>the very structure of reasoning as humans do it
I don't agree that the machine reasons even close to a human as of today. It will get better of course over time. However in some not so frequent cases, it comes close. Some times, it seems like it, but only superficially i would argue. Upon closer inspection the machine spits out non sense.
[1] Human anatomy, is very difficult to get right, like an artist. Many/all of the artists, point out the fact, that A.I. art doesn't have soul in the pictures. I share the same sentiment.
Hofstadter talks about something similar in his books.
To me it makes no sense to say that a LLM could explain its own reasoning if it does no (logical) reasoning at all. It might be able to explain how the neural network calculates its results. But there are no logical reasoning steps in there that could be explained, are there?
IANAE but although an LLM meets the definition of a Markov Chain as I understand it (current state in, probabilities of next states out), the big black box that spits out the probabilities could be doing anything.
Is it fundamentally impossible for reasoning to be an emergent property of an LLM, in a similar way to a brain? They can certainly do a good impression of logical reasoning- better than some humans in some cases?
Just because an LLM can be described as a Markov Chain doesn’t mean it _uses_ Markov Chains? An LLM is very different to the normal examples of Markov Chains I’m familiar with.
Or am I missing something?
In any case, coemu is an interesting related idea to constrain AIs to thinking in ways we can understand better:
https://futureoflife.org/podcast/connor-leahy-on-agi-and-cog...
https://www.alignmentforum.org/posts/ngEvKav9w57XrGQnb/cogni...
The LLM seems to be only one of the many building blocks and is used to supply priors / transition probabilities that are used elsewhere in downstream part of the model.
But since the markov chain becomes exponentially larger whit the amount of states this is a very nitpicky and meaningless point.
Clearly to say something its a markov chain and have that mean something you need to say the thing its doing could be more or less compressed to a simple markov chain for bigrams or something like that, but that is just not true empirically, not even for gpt2. Just this is already pretty hard to make into a reasonable size markov chain https://arxiv.org/abs/2211.00593.
Just saying that it outputs probabilities from each state is not enough, the states are english strings, there's (number of tokens)^contex_lenght possible states for a certain length that's not a reasonable markov chain that you could actually implement or run.
It's not the Creature from the Lagoon, its an engineering artifact created by engineers. I haven't heard them say it does logical deduction according to any set of logic-rules. What I've read is it uses Markov chains. That makes sense because basically an LLM given a string-input should reply with another string that is the most likely follow-up string to the first string, based on all the texts it crawled up from the internet.
If internet had lots and lots of logical reasoning statements then a LLM might be good at producing what looks like logical reasoning, but that would still be just response with the most likely follow-up string.
The reason the results of LLMs are so impressive is that at some point the quantity of the data makes a seemingly qualitative difference. It's like if you have 3 images and show them each to me one after the other I will say I saw 3 images. But if you show me thousands of images 24 per second and the images are small variations of the previous images then I say I see a MOVING PICTURE. At some point quantity becomes quality.
Of course, if you were trying to use GPT-4 to explain GPT-4 then I think the Gödel incompleteness theorem would be more relevant, and even then I'm not so sure.
There's no formal axiom system being dealt with here, afaict?
Do you just generally mean "there may be some kind of self-reference, which may lead to some kind of liar-paradox-related issues"?
Some training forms include entailment : “if A then B”. I hope this is first order logic which does have an axiom system :)
Well, what exactly would we be showing that these models can’t do? Quines exist, so there’s no general principle preventing reflection in general. We can certainly write poems (etc.) which describe their own composition. A computer can store specifications (and circuit diagrams, chip designs, etc.) for all its parts, and interactively describe how they all work.
If we are just saying “ML models can’t solve the halting problem”, then ok, duh. If we want to say “they don’t prove their own consistency” then also duh, they aren’t formal systems in a sense where “are they consistent (as a formal system)?” even makes sense as a question.
I don’t see a reason why either Gödel or Turing’s results would be any obstacle for some mechanism modeling/describing how it works. They do pose limits on how well they can describe “what they will do” in a sense of like, “what will it ‘eventually’ do, on any arbitrary topic”. But as for something describing how it itself works, there appears to be no issue.
If the task to give it was something like “is there any input which you could be given which would result in an output such that P(input,output)” for arbitrary P, then yeah I would expect such diagonalization problems to pop-up.
But a system having a kind of introspection about how it works, rather than answering arbitrary questions about its final outputs (such as, program output, or whether a statement has a proof), seems totally fine.
Side note: One funny thing: (aiui) it is theoretically possible for a oracle that can have random behavior, to act (in a certain sense) as a halting-oracle for Turing machines with access to the same oracle.
That’s not to say that we can irl construct such a thing, as we can’t even make a halting oracle for normal Turing machines. But, if you add in some random behavior for the oracles, you can kinda evade the problems that come from the diagonalization.
In a real sense, all of the future discoveries of mathematics already exist in the "training set" of our present understanding, we just haven't thought it all the way through yet. If we discover something new, can we say that the concept didn't exist, or that it "couldn't be inferred" from previous work?
I think the same would apply to LLMs and their understanding of the way we encode information using language. Given their radically different approach to understanding the same medium, they are well poised to both confirm many things we understand intuitively as well as expose the shortcomings of our human-centric model of understanding.
On the other hand, that is an incredibly high bar.
And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw inventiveness. Not a stochastic parrot looping through a large corpus of information and a table of weights on word pairings.
Has AI ever managed to learn something humans didn't already know? It's got all the physics text books in its data set. Can it make novel inferences from that? How about in math?
Language took dozens of millennia to form, and animals have long had vocalizations. Seems like a natural building on top of existing features.
> Has AI ever managed to learn something humans didn't already know?
AlphaZero invented all new categories of strategy for games like Go, when previously we thought almost all possible tactics had been discovered. AIs are finding new kinds of proteins we never thought about, which will blow up the fields of medicine and disease in a few years once the first trials are completed.
Sure, but in a simulated evolutionary algorithm, you can hit a few hundred generations in a matter of seconds.
Indeed, the identification of an abstraction, followed by a definition of that abstraction and an enshrinement of the concept in the form of a word or phrase, in and of itself, shortcuts the evolutionary path altogether. AI isn't starting from scratch: it's starting from a dictionary larger than any human alive knows and in-memory examples of humans conversing on nearly every topic imaginable.
We never thought "all possible tactics" had been discovered with Go. We quite literally understood that Go had a more complex search space than Chess, with far more possible moves and outcomes. And I don't think anyone has any kind of serious theorem that "all possible tactics" have been discovered in either game, to this day.
That being said, Go and Chess are enumerable games with deterministic, bounded complexity and state space.
The protein folding example is a neat one, I definitely think it's interesting to see what develops there. However, protein folding has been modeled by Markov State models for decades. The AlphaFold breakthrough is fantastic, but it was already known how to generate models from protein structures: it was just computationally expensive.
It was also carefully crafted by humans to achieve what it did: https://www.youtube.com/watch?v=gg7WjuFs8F4. So this is an example of humans using neural network technology that humans invented to achieve a desired solution to a known problem that they themselves conceived. The AI didn't tell us something we didn't already know. It was an expert system built with teams of researchers in the loop the whole way through.
The reason why "naming things" is the other hard problem in computer science, after cache invalidation, is that the process of identifying, creating, and describing ideal abstractions is itself inherently difficult.
EDIT: I see below you gave some examples, like invention of language before it existed, and new theorems in math that presumably would be of interest to mathematicians. Those ones are fair enough in my opinion. The AI isn't quite good enough for those ones I think, but I also think newer versions trained with only more CPU/GPU and more parameters and more data could be 'AI scientists' that will make these kinds of concepts.