Hacker News Reader

Comment by estebarb | Hacker News Reader

I'm not sure about it makes sense to apply Gödel's theorem to AI. Personally, I prefer to think about it in terms of basic computability theory:

We think, that is a fact.

Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.

Now, the question is: can we create a smaller function capable of performing the same feat?

If we assume that that function is computable in the Turing sense then, kinda yes, there are an infinite number of turing machines that given enough time will be able to produce the expected results. Basically we need to find something between our own brain and the Kolmogorov complexity limit. That lower bound is not computable, but given that my cats understands when we are discussing to take them to the vet then... maybe we don't really need a full sized human brain for language understanding.

We can run Turing machines ourselves, so we are at least Turing equivalent machines.

Now, the question is: are we at most just Turing machines or something else? If we are something else, then our own CoT won't be computable, no matter how much scale we throw at it. But if we are then it is just matter of time until we can replicate ourselves.

View on HN

Many philosophical traditions which incorporate a meditation practice emphasize that your consciousness is distinct from the contents of your thoughts. Meditation (even practiced casually) can provide a direct experience of this.

When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs. So human thinking is probably computable, and I think that LLMs can be said to be ”think” in ways that are analogous to what we do.

But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way.

I don’t necessarily think that you need to subscribe to dualism or religious beliefs to explain consciousness - it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”.

But I do think it’s still important to maintain a distinction between “thinking” (computable, we do it, AIs do it as well) and “consciousness” (we experience it, probably many animals experience it also, but it’s orthogonal to the linguistic or logical reasoning processes that AIs are currently capable of).

At some point this vague experience of awareness may be all that differentiates us from the machines, so we shouldn’t dismiss it.

> I think that LLMs can be said to be ”think” in ways that are analogous to what we do. ... But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way.

"We've all been dancing around the basic issue: does Data have a soul?" -- Captain Louvois. https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(e...

I for one (along with many thinkers) define intelligence as the extent to which an agent can solve a particular task. I choose the definition to separate it from issues involving consciousness.

Both matter of course.

And that's a useful and pragmatic definition, because it's very hard to measure the other definition even just for other humans.

That may be an illusion. And easily outputtable in the same way. Function calling in the output to release certain hormones etc.

We don’t know that LLMs generating tokens for scenarios involving simulations of conscious don’t already involve such experience. Certainly such threads of consciousness would currently be much less coherent and fleeting than the human experience, but I see no reason to simply ignore the possibility. To whatever degree it is even coherent to talk about the conscious experience of others than yourself (p-zombies and such), I expect that as AIs’ long term coherency improves and AI minds become more tangible to us, people will settle into the same implicit assumption afforded to fellow humans that there is consciousness behind the cognition.

The very tricky part then is to ask if the consciousness/phenomenological experience that you postulate still happens if, say, we were to compute the outputs of an LLM by hand… while difficult, if every single person on earth did one operation per second, plus some very complicated coordination and results gathering, we could probably predict a couple of tokens for an LLM at some moderate frequency… say, a couple of tokens a month? a week? A year? A decade? Regardless… would that consciousness still have an experience? Or is there some threshold of speed and coherence, or coloration that would be missing and result in failure for it to emerge?

Impossible to answer.

Btw I mostly think it’s reasonable to think that there might be consciousness, phenomenology etc are possible in silicon, but it’s tricky and unverifiable ofc.

> would that consciousness still have an experience?

If the original one did, then yes, of course. You're performing the exact same processing.

Imagine if instead of an LLM the billions of people instead simulated a human brain. Would that human brain experience consciousness? Of course it would, otherwise they're not simulating the whole brain. The individual humans performing the simulation are now comparable to the individual neurons in a real brain. Similarly, in your scenario, the humans are just the computer hardware running the LLM. Apart from that it's the same LLM. Anything that the original LLM experiences, the simulated one does too, otherwise they're not simulating it fully.

Yes that’s my main point - if you accept the first one, then you should accept the second one (though some people might find the second so absurd as to reject the first).

> Imagine if instead of an LLM the billions of people instead simulated a human brain. Would that human brain experience consciousness? Of course it would, otherwise they're not simulating the whole brain.

However, I don’t really buy “of course it would,” or in another words the materialist premise - maybe yes, maybe no, but I don’t think there’s anything definitive on the matter of materialism in philosophy of mind. as much as I wish I was fully a materialist, I can never fully internalize how sentience can uh emerge from matter… in other words, to some extent I feel that my own sentience is fundamentally incompatible with everything I know about science, which uh sucks, because I definitely don’t believe in dualism!

It would certainly with sufficient accuracy honestly say to you that it's conscious and believes it whole heartily, but in practice it would need to a priori be able describe external sense data, as it's not separate necessarily from the experiences, which intrinsically requires you to compute in the world itself otherwise it would only be able to compute on, in a way it's like having edge compute at the skins edge. The range of qualia available at each moment will be distinct to each experiencer with the senses available, and there likely will be some overlap in interpretation based on your computing substrate.

We in a way can articulate the underlying chemputation of the universe mediated through our senses, reflection and language, turn a piece off (as it is often non continuous) and the quality of the experience changes.

But do you believe in something constructive? Do you agree with Searle that computers calculate? But then numbers and calculation are immaterial things that emerge from matter?

You are assuming that consciousness can be reproduced by simulating the brain. Which might be possible but it's by no means certain.

You can simulate as much of the human as you need to. So long as consciousness is a physical process (or an emergent property of a physical process), it can be simulated.

The notion that it is not a physical process is an extraordinary claim in its own right, which itself requires evidence.

Is simulate the right word there? With a hundred trillion connections between 80 billion neurons, it seems unlikely that it would ever be worth simulating a human brain, because it would be simpler to just build one than to assemble a computer complex enough to simulate it.

You can simulate as much of an aircraft as you need to. So long as flying is a physical process, it can be simulated.

But your simulation will never fly you over an ocean, it will never be an aircraft or do what aircraft do. A simulation of heat transfer will not cook your dinner. A simulation of Your assumption that a simulation of a mind is a mind, requires evidence.

> But your simulation will never fly you over an ocean

It will fly over a simulated ocean just fine. It does exactly what aircraft do, within the simulation. By adding “you” to the sentence you've made it an apples to oranges comparison because “you” is definitionally not part of the simulation. I don't see how you could add the same “you” to “it will simulate consciousness just fine”.

> "It does exactly what aircraft do"

It doesn't move real Oxygen and Nitrogen atoms, it doesn't put exhaust gas into the air over the ocean, it doesn't create a rippling sound and pressure wave for a thousand miles behind it, it doesn't drain a certain amount of jet fuel from the supply chain or put a certain amount of money in airline and mechanics' pockets, it doesn't create a certain amount of work for air traffic controllers... reductio ad abusurdum is that a flipbook animation of a stickman aircraft moving over a wiggly line ocean is a very low granularity simulation and "does exactly what aircraft do" - and obviously it doesn't. No amount of adding detail to the simulation moves it one inch closer to doing 'exactly what aircraft do'.

> "I don't see how you could add the same “you” to “it will simulate consciousness just fine”"

by the same reductio-ad-absurdum I don't see how you can reject a stickman with a speech bubble drawn over his head as being "a low granularity simulated consciousness". More paper, more pencil graphite, and the stickman will become conscious when there's enough of it. Another position is that adding things to the simulation won't simulate consciousness just fine - won't move it an inch closer to being conscious; it will always be a puppet of the simulator, animated by the puppeteer's code, always wooden Pinocchio and never a real person. What is the difference between these two:

a) a machine with heat and light and pressure sensors, running some code, responding to the state of the world around it.

b) a machine with heat and light and pressure sensors, running some code [converting the inputs to put them into a simulation, executing the simulation, converting the outputs from the simulation], and using those outputs to respond to the state of the world around it.

? What is the 'simluate consciousness' doing here at all, why is it needed? To hide the flaw in the argument; it's needed to set up the "cow == perfectly spherical massless simulated cow" premise which makes the argument work in English words. Instead of saying something meaningful about consciousness, one states that "consciousness is indistinguishable from perfectly spherical massless simulated consiousness" and then states "simply simulate it to as much detail as needed" and that allows all the details to be handwaved away behind "just simulate it even more (bro)".

Pointing out that simulations are not the real thing is the counter-argument. Whether or not the counter-argument can be made by putting "you" into a specific English sentence is not really relevant, that's only to show that the simulated aircraft doesn't do what the real aircraft does. A simulated aircraft flying over a simulated ocean is no more 'real' than drawing two stick figures having a conversation in speech bubbles.

You need to clearly define what constitutes "real" before we can meaningfully talk about the distinction between "real" atoms and simulated ones.

As far as physics go, it's all just numbers in the end. Indeed, the more we keep digging into the nature of reality, the more information theory keeps popping up - see e.g. the holographic principle.

> "As far as physics go, it's all just numbers in the end."

No it isn't; numbers are a map, maps are not the territory. You are asking me to define how a map is different from a city, but you are not accepting that the city is made of concrete and is square kilometers large and the map is made of paper and is square centimeters large as a meaningful difference, when I think it's such an obvious difference it's difficult to put any more clearly.

What constitutes a real atom: a Hydrogen atom capable of combining with Oxygen to make water, capable of being affected by the magnetic field of an MRI scanner, etc.

What constitutes a simulated atom: a pattern of bits/ink/numbers which you say "this is a representation of a Hydrogen atom", capable of nothing, except you putting some more bits/ink/numbers near it and speaking the words "this is it interacting to make simulated water".

Ok, you are saying that a map is different than the territory. That a simulation is meaningfully different.

Do you deny that you could be in a simulation right now, in the matrix? What you actually think are are molecules of oxygen are actually simulated molecules. That there is no way for you to every tell the difference.

You just wrote a lot of text just to say that you don't accept the simulation as “real”.

That's just semantics. I'm not here to argue what the word “real” means. Of course you can define it in such a way that the simulated aircraft isn't “really” flying over an ocean, and it would be just as valid as any other definition, but it doesn't say anything meaningful or insightful about the simulation.

Nobody contests your point that the simulated aircraft isn't going over a real ocean and isn't generating work for real-life air traffic controllers. But conversely you don't seem to contest the claim that oceans and air traffic controllers could be simulated, too. Therefore, consciousness can be simulated as well, and it would be a simulated consciousness that just doesn't fall into your definition of “real”.

> It's very difficult to find some way of defining rather precisely something we can do that we can say a computer will never be able to do. There are some things that people make up that say that, "While it's doing it, will it feel good?" or, "While it's doing it, will it understand what it's doing?" or some other abstraction. I rather feel that these are things like, "While it's doing it, will it be able to scratch the lice out of it's hair?" No, it hasn't got any hair nor lice to scratch from it, okay?

> You've got to be careful when you say what the human does, if you add to the actual result of his effort some other things that you like, the appreciation of the aesthetic... then it gets harder and harder for the computer to do it because the human beings have a tendency to try to make sure that they can do something that no machine can do. Somehow it doesn't bother them anymore, it must have bothered them in earlier times, that machines are stronger physically than they are...

- Feynman

https://www.youtube.com/watch?v=ipRvjS7q1DI

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra

Maybe we can swap out "think" with "experience consciousness"

You need to define "consciousness" first for the question to have any meaning, but all our definitions of consciousness seem to ultimately boil down to, "this thing that I'm experiencing".

What about the famous solution provided by Descartes, “Cogito ergo sum”? Let's assume the fact that “we think”, so we can put it in a function to be computable, how is that going to prove that “I exist” for a machine? How is the machine going to perceive itself as a conscious being?

What we have and AI don't are emotions. After all, that what animates us to survive and reproduce. Without emotions we can't classify and therefore store our experiences because there no reason to remember something which we are indifferent about. This includes everything not accessible by our senses. Our abilities are limited to what is needed for survival and reproduction because all the rest would consume our precious resources.

The larger picture is that our brains are very much influenced by all the chemistry that happens around our units of computation (neurones); especially hormones. But (maybe) unlike consciousness, this is all "reproducible", meaning it can be part of the algorithm.

> When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs.

Function can mean inputs-outputs. But it can also mean system behaviors.

For instance, recurrence is a functional behavior, not a functional mapping.

Similarly, self-awareness is some kind of internal loop of information, not an input-output mapping. Specifically, an information loop regarding our own internal state.

Today's LLMs are mostly not very recurrent. So might be said to be becoming more intelligent (better responses to complex demands), but not necessarily more conscious. An input-output process has no ability to monitor itself, no matter how capable of generating outputs. Not even when its outputs involve symbols and reasoning about concepts like consciousness.

So I think it is fair to say intelligence and consciousness are different things. But I expect that both can enhance the other.

Meditation reveals a lot about consciousness. We choose to eliminate most thought, focusing instead on some simple experience like breathing, or a concept of "nothing".

Yet even with this radical reduction in general awareness, and our higher level thinking, we remain aware of our awareness of experience. We are not unconscious.

To me that basic self-awareness is what consciousness is. We have it, even when we are not being analytical about it. In meditation our mind is still looping information about its current state, from the state to our sensory experience of our state, even when the state has been reduced so much.

There is not nothing. We are not actually doing nothing. Our mental resting state is still a dynamic state we continue to actively process, that our neurons continue to give us feedback on, even when that processing has been simplified to simply letting that feedback of our state go by with no need to act on it in any way.

So consciousness is inherently at least self-awareness in terms of internal access to our own internal activity. And that we retain a memory of doing this minimal active or passive self-monitoring, even after we resume more complex activity.

My own view is that is all it is, with the addition of enough memory of the minimal loop, and a rich enough model of ourselves, to be able to consider that strange self-awareness looping state afterwards. Ask questions about its nature, etc.

This is what I wrote while I was thinking about the same topic before I can across your excellent comment; as if it’s a summary of what you just said:

Consciousness is nothing but the ability to have internal and external senses, being able to enumerate them, recursively sense them, and remember the previous steps. If any of those ingredients are missing, you cannot create or maintain consciousness.

Thanks. I do believe that is a good summary of what I was saying.

LLMs are recurrent in the sense that you describe, though, since every token of output they produce is fed back to them as input. Indeed, that is why reasoning models are possible in the first place, and it's not clear to me why the chain-of-thought is not exactly that kind of "internal loop of information" that you mention.

> Meditation reveals a lot about consciousness. We choose to eliminate most thought, focusing instead on some simple experience like breathing, or a concept of "nothing".

The sensation of breathing still constitutes input. Nor is it a given that a thought is necessarily encodeable in words, so "thinking about concept of nothing" is still a thought, and there's some measurable electrochemical activity encoding that in the brain which encodes it. In a similar vein, LLMs deal with arbitrary tokens, which may or may not encode words - e.g. in multimodal LMs, input includes tokens encoding images directly without any words, and output can similarly be non-word tokens.

> chain-of-thought is not exactly that kind of "internal loop of information" that you mention.

It is, but (1) the amount of looping in models today is extremely trivial. if our awareness loop is on the order of milliseconds, we experience it on the order of thousands of milliseconds at a minimum. And consider and consolidate our reasoning about experiences over minutes, hours, even days. Which would be thousands to many millions of iterations of experiential context.

Then (2), the looping of models today is not something the model is aware of at a higher level. It processes the inputs iteratively, but it isn't able to step back and examine its own responses recurrently at a second level in a different indirect way.

Even though I do believe models can reason about themselves and behave as if they did have that higher functionality.

But their current ability to reason like that has been trained into them by human behavior, not learned independently by actually monitoring their own internal dynamics. They cannot yet do that. We do not learn we are conscious, or become conscious, by parroting others conscious enabled reasoning. A subtle but extremely important difference.

Finally, (3) they don't build up a memory of their internal loops, much less a common experience from a pervasive presence of such loops.

Those are just three quite major gaps.

But they are not fundamental gaps. I have no doubt that future models will become conscious as limitations are addressed.

When I was a kid, I used to imagine if that society ever developed AI, there would be widespread pushback to the idea that computers could ever develop consciousness.

I imagined the Catholic Church, for example, would be publishing missives reminding everyone that only humans can have souls, and biologists would be fighting an quixotic battle to claim that consciousness can arise from physical structures and forces.

I'm still surprised at how credulous and accepting societies have been of AI developments over the last few years.

Probably because we've been conditioned to accept that machines, no matter how friendly, are not really conscious in the way we are, so there is no risk of them needing to be treated differently than a hammer.

AI developments over the last few years have not needed that view to change.

To state it's a turing machine might be a bit much but there might be a map between substrates to some degree, and computers can have a form of consciousness, an inner experience, basically the hidden layers and clearly the input of senses, but it wouldn't be the same qualia as a mind, I suspect it has more to due with chemputation and is dependent on the substrate doing the computing as opposed to a facility thereof, up to some accuracy limit, we can only detect light we have receptors for after all. To have qualia distinct to another being you need to compute on a substrate that can accurately fool the computation, fake sugar instead of sugar for example.

>it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”.

I've heard this idea before but I have never been able to make head or tail of it. Consciousness can't be an illusion, because to have an illusion you must already be conscious. Can a rock have illusions?

I think it’s more apt to say that free will is an illusion.

Well, it entirely depends on how you even define free will.

Btw, Turing machines provide some inspiration for an interesting definition:

Turing (and Gödel) essentially say that you can't predict what a computer program does: you have to run it to even figure out whether it'll halt. (I think in general, even if you fix some large fixed step size n, you can't even predict whether an arbitrary program will halt after n steps or not, without essentially running it anyway.)

Humans could have free will in the same sense, that you can't predict what they are doing, without actually simulating them. And by an argument implied by Turing in his paper on the Turing test, that simulation would have the same experience as the human would have had.

(To go even further: if quantum fluctuations have an impact on human behaviour, you can't even do that simulation 100% accurately, because of the no cloning theorem.

To be more precise: I'm not saying, like Penrose, that human brains use quantum computing. My much weaker claim is that human brains are likely a chaotic system, so even a very small deviation in starting conditions can quickly lead to differences in outcome.

If you are only interested in approximate predictions, identical twins show that just getting the same DNA and approximation of the environment gets you pretty far in making good predictions. So cell level scans could be even better. But: not perfect.)

> Humans could have free will in the same sense, that you can't predict what they are doing, without actually simulating them.

I think it's a good point, but I would argue it's even more direct than that. Humans themselves can't reliably predict what they are going to do before they do it. That's because any knowledge we have is part of our deliberative decision-making process, so whenever we think we will do X, there is always a possibility that we will use that knowledge to change our mind. In general, you can't feed a machine's output into its input except for a very limited class of fixed point functions, which we aren't.

So the bottom line is that seen from the inside, our self-model is a necessarily nondeterministic machine. We are epistemically uncertain about our own actions, for good reason, and yet we know that we cause them. This forms the basis of our intuition of free will, but we can't tell this epistemic uncertainty apart from metaphysical uncertainty, hence all the debate about whether free will is "real" or an "illusion". I'd say it's a bit of both: a real thing that we misinterpret.

You are right about the internal model, but I wouldn't dismiss the view from the outside.

Ie I wouldn't expect humans without free will to be able to predict themselves very well, either. Exactly as you suggest: having a fixed point (or not) doesn't mean you have free will.

The issue I have with the view from the outside is that it risks leading to a rather anthropomorphic notion of free will, if the criterion boils down to that an entity can only have free will if we can't predict its behavior.

I'm tempted to say an entity has free will if it a) has a self-model, b) uses this self-model as a kind of internal homunculus to evaluate decision options and c) its decisions are for the most part determined by physically internal factors (as opposed as external constraints or publicly available information). It's tempting to add a threshold of complexity, but I don't think there's any objectively correct way to define one.

I don't understand why a self-model would be necessary for free will?

> [...] c) its decisions are for the most part determined by physically internal factors (as opposed as external constraints or publicly available information).

I don't think humans reach that threshold. Though it depends a lot on how you define things.

But as far as I can tell, most of my second-to-second decisions are very much coloured by the fact that we have gravity and an atmosphere at comfortable temperatures (external factors), and if you changed that all of a sudden, I would decide and behave very differently.

> It's tempting to add a threshold of complexity, but I don't think there's any objectively correct way to define one.

Your homunculus is one hell of a complexity threshold.

There is no you to have the illusion.

> We think, that is a fact.

It likely is a fact, but we don't really know what we mean by "think".

LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not.

We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking.

Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.

I don't think its useful or even interesting to talk about AI in relation to how humans think, or whether or not they will be "conscious" whatever that might mean.

AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain.

The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them.

When will it be able to set its own goals, and know when it has achieved them?

When will it be able to recognize that it doesn't know something and do the work to fill in the blanks.

I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.

There was a study where they trained a model to lie. When they looked at what was happening internally they could see it knew the truth but just switched things at the outer nodes to lie.

I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models in order to bang out a snarky comment is probably trying to sell you something (and crypto’s likely involved.)

I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.

I've never had any time for Penrose. Gödel’s theorem "merely" asserts that in any system capable of a specific form of expression there are statements which are true but not provable. What this has to do with (a) limits to computation or (b) human intelligence has never been clear to me, despite four decades or more of interest in the topic. There's no reason I can see why we should think that humans are somehow without computational limits. Whether our limits correspond to Gödel’s theorem or not is mildly interesting, but not really foundational from my perspective.

At the end of the day Penrose's arguments is just Dualism.

Humans have a special thingy that makes the consciousness Computers do not have the special thingy Therefore Computers cannot be consciousness.

But Dualism gets you laughed at these days so Dualists have to code their arguments and pretend they aren't into that there Dualism.

Penrose's arguments against AI has always felt to me like special pleading that humans (or to stretch a bit further, carbon based lifeforms) are unique.

While I don't like Penrose's argument and I think it stands on very shaky ground, I very much disagree it's a form of dualism. His argument is simply that human thinking is not reducible to a Turing machine, that it is a form of hyper-computation.

If this were to be true, it would follow that computers as we build them today would fundamentally not be able to match human problem-solving. But it would not follow, in any way, that it would be impossible to build "hyper computers" that do. It just means you wouldn't have any chance of getting there with current technology.

Now, I don't think Penrose's arguments for why he thinks this is the case are very strong. But they're definitely not mystical dualistic arguments, they're completely materialistic mathematical arguments. I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.

> I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.

That was my understanding on Penrose's position as well which is just a "Consciousness of the Gaps" argument. As we learn more about quantum operations the space for Consciousness as a special property of humans disapears.

Penrose doesn’t think that consciousness is special to humans. He thinks most animals have it and more importantly to your point, he thinks that there is no reason that we won’t someday construct artificial creations that have it.

I just watched an interview where he made that exact statement nearly word for word.

His only argument is that it is not computable, not that it’s not physical. He does think the physical part involves the collapse of the wave function due to gravity, and that somehow the human brain is interacting with that.

So to produce conciseness in his view, you’d need to construct something capable of interacting with the quantum world the same way he believes organic brains do (or something similar to it). A simulation of the human brain wouldn’t do it.

He proposed a proof of platonism: Mandelbrot set has a stable form and is not subjective, because it doesn't fit in human mind due to its sheer complexity, consequently it exists objectively. His beliefs are pretty transparent.

> I think we have a pretty good idea that we are not stochastic parrots

what if we are?

and our brain is the "billion parameter model", continuously "trining", that takes input and spits out output

I understand we as a species might be reluctant to admit that we are just matter and our thinking/consciousness is just electricity flowing

btw, I'm not selling anything :)

> I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models

On the contrary, we have 86B neurons in the brain, the weighting of the connections is the important thing, but we are definitely 'running' a model with many billions of parameters to produce our output.

The theory by which the brain mainly works by predicting the next state is called predictive coding theory, and I would say that I find it pretty plausible. At the very least, we are a long way from knowing for certain that we don't work in this way.

Try taking any of the LLM models we have, and making it learn (adjust its weights) based on every interaction with it. You'll see it quickly devolves into meaninglessness. And yet we know for sure that this is what happens in our nervous system.

However, this doesn't mean in any way that an LLM might not produce the same or even superior output than a human would in certain very useful circumstances. It just means it functions fundamentally differently on the inside.

Maybe this is just a conversation about what "fundamentally differently" means then.

Obviously the brain isn't running an exact implementation of the attention paper, and your point about how the brain is more malleable than our current llms is a great point, but that just proves they aren't the same. I fully expect that future architectures will be more malleable, if you think that such hypothetical future architectures will be fundamentally different from the current ones then we agree..

> On the contrary, we have 86B neurons in the brain

The neurons (cells) in even a fruit flies brain are orders of magnitude more complex than the "neurons" (theoretical concept) in a neural net.

> the weighting of the connections is the important thing

In a neural net, sure.

In a biological brain, many more factors are important: The existence of a pathway. Antagonistic neurotransmitters. NT re-incorporation. NT-binding sensitivity. Excitation potential. Activity of Na/K channels. Moderating enzymes.

Even what we last ate or drank, how rested, old, hydrated, we are, when our lats physical activity took place, and all the interactions prior to an input influence how we analyse and integrate it.

> but we are definitely 'running' a model with many billions of parameters to produce our output.

No, we are very definitely not. Many of our mental activities have nothing to do with state prediction at all.

We integrate information.

We exist as a conscious agent in the world. We interact, and by doing so change our own internal state alongside the information we integrate. We are able to, from this, simulate our own actions and those of other agents, and model the world around us, and then model how an interaction with that world would change the model.

We are also able to model abstract concepts both in and outside the world.

We understand what concepts, memories, states, and information mean both as abstract concepts and concrete entities in the universe.

We communicate with other agents, simultaneously changing their states and updating our modeling of their internal state (theory of the mind, I know that you know that I know, ...)

We filter, block, change, and create information.

And of course we constantly learn and change the way we do ALL OF THIS, consciously and subconsciously.

> At the very least, we are a long way from knowing for certain that we don't work in this way.

https://en.wikipedia.org/wiki/Russell%27s_teapot

OK, let me be more clear, because I'm not sure what you're arguing against.

If the process in the brain is modellable at all, then it is certainly a model with at a minimum many billions of parameters. Your list of additional parameters if anything supports that rather than arguing against it. If you want to argue with that contention, I think you need to argue that the process isn't modellable, which if you want to talk about burden of proof, would place a huge burden on you. But maybe I misunderstood you. I thought you were saying that it's ludicrous to say we're using as many as billions of parameters, but perhaps you're trying to say that billions is obviously far too small, in which case I agree.

My second point, which is that there's a live theory that prediction may be a core element of our consciousness was intended as an interesting aside, I don't know how it will stand the test of time and I certainly don't know if its correct or not, I intended only to use it to prove that the things you seem to think are obvious are not in fact obvious to everyone.

For example, that big list of things that you are using as an argument against prediction doesn't work at all because you don't know whether they are implemented via a predictive process in the brain or not.

It feels that rather than arguing against modellability or large numbers of parameters or prediction you're arguing against the notion that the human brain is exactly an llm, which is an idea so obviously true I don't think anyone actually disagrees with it.

> Your list of additional parameters if anything supports that rather than arguing against it.

> perhaps you're trying to say that billions is obviously far too small, in which case I agree.

No, it doesn't, and I don't.

The processes that happen in a living brain don't just map to "more params". It doesn't matter how many learnable parameters you have...unless you actually change the paradigm, an LLM or similar construct is incapable of mapping a brain, period. The simple fact that the brains internal makeup is itself changeable, already prevents that.

> prediction may be a core element of our consciousness

No it isn't, and it's trivially easy to show that.

Many meditative techniques exist where people "empty their mind". They don't think or predict anything. Does that stop consiousness? Obviously not.

Can we do prediction? Sure. Is it a "core element", aka. indispensable for consciousness? No.

I am not a neuroscientist, but I think it's likely that LLMs (with 10s/100s of billions of parameters) and the human brain (with 1-2 orders of magnitude more neural connections[1]) process language in analogous ways. This process is predictive, stochastic, sensitive to constantly-shifting context, etc. IMO this accounts for the "unreasonable effectiveness" of LLMs in many language-related tasks. It's reasonable to call this a form of intelligence (you can measure it, solve problems with it, etc).

But language processing is just one subset of human cognition. There are other layers of human experience like sense-perception, emotion, instinct, etc. – maybe these things could be modeled by additional parameters, maybe not. Additionally, there is consciousness itself, which we still have a poor understanding of (but it's clearly different from intelligence).

So anyway, I think that it's reasonable to say that LLMs implement one sub-set of human cognition (the part that has to do with how we think in language), but there are many additional "layers" to human experience that they don't currently account for.

Maybe you could say that LLMs are a "model distillation" of human intelligence, at 1-2 orders of magnitude less complexity. Like a smaller model distilled from a larger one, they are good at a lot of things but less able to cover edge cases and accuracy/quality of thinking will suffer the more distilled you go.

We tend to equate "thinking" with intelligence/language/reason thanks to 2500 years of Western philosophy, and I believe that's where a lot of confusion originates in discussions of AI/AGI/etc.

[1]: https://medicine.yale.edu/lab/colon-ramos/overview/#:~:text=...

>I am not a neuroscientist, but I think it's likely that LLMs (with 10s of billions of parameters) and the human brain (with 1-2 orders of magnitude more neural connections[1]) process language in analogous ways

Related is the platonic representation hypothesis where models apparently converge to similar representations of relationships between data points.

https://phillipi.github.io/prh/ https://arxiv.org/abs/2405.07987

Interesting. I'm not sure I'd use the term "Platonic" here, because that tends to have implications of mathematical perfection / timelessness / etc. But I do think that the corpuses of human language that we've been feeding to these models contain within them a lot of real information about the objective world (in a statistical, context-dependent way as opposed to a mathematically precise one), and the AIs are surfacing this information.

To put this another way, I think that you can say that much of our own intelligence as humans is embedded in the sum total of the language that we have produced. So the intelligence of LLMs is really our own intelligence reflected back at us (with all the potential for mistakes and biases that we ourselves contain).

Edit: I fed Claude this paper, and "he" pointed out to me that there are several examples of humans developing accurate conceptions of things they could never experience based on language alone. Most readers here are likely familiar with Helen Keller, who became an accomplished thinker and writer in spite of being blind and deaf from infancy (Anne Sullivan taught her language despite great difficulty, and this Keller's main window to the world). You could also look at the story of Eşref Armağan, a Turkish painter who was blind from birth – he creates recognizable depictions of a world that he learned about through language and non-visual senses).

> there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.

We mistakenly assume, they are true because perhaps we want them to be true. But we have no proof that either of these are true.

Descartes disagrees.

And? Mind-body dualism as Descartes imagined has been practically disproven on almost every front.

Mind-body dualism has nothing to do with this. The point is that, as Descartes observed, the fact that I myself am thinking proves that I exist. This goes directly against what northern-lights said, when he said that we have no proof that reasoning exists or that we do it.

“Cogito ergo cogito”?

Kant addressed this Cartesian duality in the "The paralogisms of pure reason" section of the Transcendental Dialectic within his Critique of Pure Reason. He points out that the "I" in "I think, therefore I am" is a different "I" in the subject part vs the object part of that phrase.

Quick context: His view of what constitutes a subject, which is to say a thinking person in this case, is one which over time (and time is very important here) observes manifold partial aspects about objects through perception, then through apprehension (the building of understanding through successive sensibilities over time) the subject schematizes information about the object. Through logical judgments, from which Kant derives his categories, we can understand the object and use synthetic a priori reasoning about the object.

So for him, the statement "I am" means simply that you are a subject who performs this perception and reasoning process, as one's "existence" is mediated and predicated on doing such a process over time. So then "I think, therefore I am" becomes a tautology. Assuming that the "I" in "I am" exists as an object, which is to say a thing of substance, one which other thinking subjects could reason about, becomes what he calls "transcendental illusion", which is the application of transcendental reasoning not rooted in sensibility. He calls this metaphysics, and he focuses on the soul (the topic at hand here), the cosmos, and God as the three topics of metaphysics in his Transcendental Dialectic.

I think that in general, discussion about epistemology with regard to AI would be better if people started at least from Kant (either building on his ideas or critical of them), as his CPR really shaped a lot of the post-Enlightenment views on epistemology that a lot of us carry with us without knowing. In my opinion, AI is vulnerable to a criticism that empiricists like Hume applied to people (viewing people as "bundles of experience" and critiquing the idea that we can create new ideas independent of our experience). I do think that AI suffers from this problem, as estimating a generative probability distribution over data means that no new information can be created that is not simply a logically ungrounded combination of previous information. I have not read any discussion of how Kant's view of our ability to make new information (application of categories grounded by our perception) might influence a way to make an actual thinking machine. It would be fascinating to see an approach that combines new AI approaches as the way the machine perceives information and then combines it with old AI approaches that build on logic systems to "reason" in a way that's grounded in truth. The problem with old AI is that it's impossible to model everything with logic (the failure of logical posivitism should have warned them), however it IS possible to combine logic with perception like Kant proposed.

I hope this makes sense. I've noticed a lack of philosophical rigor around the discussion of AI epistemology, and it feels like a lot of American philosophy research, being rooted in modern analytical tradition that IMO can't adapt easily to an ontological shift from human to machine as the subject, hasn't really risen to the challenge yet.

This critique misses the point of Descartes. It can be reformulated as something like "a thought has happened, therefore we can know at least that something that thinks exists." Getting caught up in the subject-object semantics has no bearing on Descartes approach to objectivity. This is no more tautological than seeing a car and then concluding that cars exist.

How do you define "exists"?

Remember, this is about Cartesian duality (mind-brain duality), so the key question here is not whether a brain exists, but whether the mind exists independently of it.

Worth pointing out that we aren't Turing equivalent machines - infinite storage is not a computability class that is realizable in the universe, so far as we know (and such a claim would require extraordinary evidence).

As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Is there a Turing machine that can output all the free parameters of the standard model?

> Is there a Turing machine that can output all the free parameters of the standard model?

Sure, just hard code them.

> As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values.

Things seem to be quantised at a low enough level.

Also: interestingly enough quantum mechanics is both completely deterministic and linear. That means even if it was continuous, you could simulate it to an arbitrary precision without errors building up chaotically.

(Figuring out how chaos, as famously observed in the weather, arises in the real world is left as an exercise to the reader. Also a note: the Copenhagen interpretation introduces non-determinism to _interpret_ quantum mechanics but that's not part of the underlying theory, and there are interpretations that have no need for this crutch.)

> Things seem to be quantised at a low enough level.

Some things. But others are very much not: in particular, space and time are not quantized, and in fact are not even quantizable. A theory in which there is some discrete minimal unit of space (or of spacetime) is trivially incompatible with special relativity, so it is incompatible with quantum mechanics (QFT, specifically).

This is easy to see from the nature of the Lorenz transformation: if two objects are at a distance D = n*min for some observer, they will be at a distance D' = n*min*gamma for some other observer, where gamma is always < 1 for an o server moving at a higher speed in a direction aligned with the two objects. So the distance for that second observer will be a non-integer multiple of the minimum distance, so your theory is no longer quantized.

We do not know if spacetime is quantized or not, and there are theories claiming that it is (LQG etc). And sure, we don't have a coherent "theory of everything" that includes quantized spacetime, and the models that we do have are contradictory if spacetime is quantized. But we already know that those models are deficient.

Our current theories only work if spacetime is not quantized. Any theory that tries to quantize spacetime needs to somehow replace special relativity and its Lorrentz transform with something completely different, while still remaining consistent with the huge number of observations that confirm SR works - especially the extremely precise experiments that confirmed QFT. This is one of the reasons why LQG is almost certainly wrong, by the way.

Note that this is separate from the problem with GR-QFT inconsistencies. All of our current theories are based on and only work if spacetime is continuous. While it's not impossible that a new theory with quantized spacetime could exist and work, it's not at all required.

The one thing about spacetime that we do believe might be quantizable, and would have to be quantized for GR and QFT to be compatible, is the curvature of spacetime. But even if spacetime can only be curved in discrete quanta, that would not mean that position would be quantized.

> Sure, just hard code them.

That would be super lucky if possible - almost all reals are not computable. How would we initialize or specify this Turing machine? Going to use non-constructive methods?

Given how quickly reality unfolds, it's a bit of a stretch to assume that "simulate it to arbitrary precision" means "computable in a digital representation in real time." I mean, if we have Turing machines that did each computation step n in 2^{-n} time.

> That would be super lucky if possible - almost all reals are not computable. How would we initialize or specify this Turing machine? Going to use non-constructive methods?

The standard model doesn't use arbitrary reals. All the parameters are rational numbers with finite precision.

Obviously, your Turing machine can only hard code a finite amount of information about the parameters, eg whatever finite prefix of their decimal expansion is known.

Btw, the speed of light is one of those parameters that's 'known' with absolute precision thanks to some clever definitions. We can do similar tricks with many of the other parameters, too.

I don’t think that’s true, there are parameters that must be measured experimentally: https://spinor.info/weblog/?p=6355 (I am not a physicist, so I welcome education to clarify this).

  > Personally, I prefer to think about it in terms of basic computability theory:

Gödel's incompleteness theorem applies to computing. I'm sure you're familiar with the Halting Problem. Gödel's applies to any axiomatic system. The trouble is, it's very hard to make a system without axioms. They are sneaky and it's different than any logic you're probably familiar with.

And don't forget Church-Turing, Gödel Numbers, and all the other stuff. Programming is math and Gödel did essential work on the theory of computation. It would be weird NOT to include his work in this conversation.

  > are we at most just Turing machines or something else?

But this is a great question. Many believe no. Personally I'm unsure, but lean no. Penrose is a clear no but he has some wacky ideas. Problem is, it's hard to tell a bad wacky idea from a good wacky idea. Rephrasing Clarke's Second Law: Genius is nearly indistinguishable from insanity. The only way to tell is with time.

But look into things like NARS and Super Turing machines (Hypercomputation). There's a whole world of important things that are not often discussed when it comes to the discussion of AGI. But for those that don't want to dig deep into the math, pick up some Sci-Fi and suspend your disbelief. Star Trek, The Orville and the like have holographic simulations and I doubt anyone would think they're conscious, despite being very realistic. But The Doctor in Voyager or Isaac in The Orville are good examples of the contrary. The Doctor is an entity you see become conscious. It's fiction, but that doesn't mean there aren't deep philosophical questions. Even if they're marked by easy to digest entertainment. Good stories are like good horror, they get under your skin, infect you, and creep in

Edit:

I'll leave you with another question. Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?

> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?

Another question. How do you go about detecting whether consciousness has arisen?

It's a great question. Best we got is ~~RLHF~~ Potter's definition: I know it when I see it

There's no evidence that hypercomputation is anything that happens in our world, is there? I'm fairly confident of the weaker claim that there's no evidence of hypercomputation in any biological system. (Who know what spinning, charged black holes etc are doing?)

> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?

A Turing machine can in principle beat the Turing test. But so can a giant lookup table, if there's any finite time limit (however generous) placed on the test.

The 'magic' would in the implementation of the table (or the Turing machine) into something that can answer in a reasonable amount of time and be physically realised in a reasonable amount of space.

Btw, that's an argument from Scott Aaronson's https://www.scottaaronson.com/papers/philos.pdf

  > There's no evidence that hypercomputation

I'm not the right person to ask this tbh. As far as I know it doesn't even well in theory. (Though I don't think they would make it useless to study)

  > can in principle beat the Turing test

What's the relevance of the Turing test? It's been beaten for over half a century.

> What's the relevance of the Turing test? It's been beaten for over half a century.

I would be very interested if you have any sources on anyone beating the Turing test in anything close to Turing's original adversarial formulation.

  ELIZA
  PARRY
  Eugene Goostman
  Mitsuku (Kuki AI)

Or see the Loebner Prize where judges aren't just average people but experts (so harder).

The Turing Test was never about intelligence or thinking, even said so by Turing himself. He made the test because those words are too vague. He specifically wanted to shift the conversation to task completion, since that is actually testable. Great! That's how science works! (Looking at you String Theory...) But science also progresses. We know more about what these things intelligence and thinking are. These still are not testable in a concrete sense, but we have better proxies than the Turing Test now.

The problem is that knowledge percolates through society slowly and with a lag. We've advanced a lot since then. I'm sure you are likely willing to believe that most LLMs these days can pass it. The Turing Test was a great start. You gotta start somewhere. But to think we came up with good tests at the same time we invented electronic computers, should give you surprise. Because it would require us to have been much smarter then than we are now.

> I would be very interested if you have any sources on anyone beating the Turing test in anything close to Turing's original adversarial formulation.

Eliza never won that adversarial version even against layman.

In what sense did Eliza ever 'win' any Turing test?

> I'm sure you are likely willing to believe that most LLMs these days can pass it.

No, I haven't seen any evidence of that.

To repeat: I am interested in evidence that any non-human can beat the Turing test in the original form given in Turing's paper, where you have the judge (human), and two contestants A and B. One of the contestants is a computer, one is a human. Everyone can see what everyone else is writing, and the human contestant can help the human judge. (But the computer can try to fake that 'helping', too.)

Turing specifically wrote: "The object of the game for the third player (B) is to help the interrogator."

I can believe that Eliza has occasionally fooled some random humans, but I can't believe Eliza managed to fool anyone when a third party was around to point out her limitations. (Especially since Eliza ain't smart enough to retaliate and fabricate some 'obvious computer limitations' to accuse the third party of.)

Most LLMs today still have some weaknesses that are easy to point out, if you let your contestants (both kinds) familiarise themselves with both humans and the LLMs in question at their leisure before the test starts.

Just for fun, I just tried out Kuki AI, and it's not going to fool anyone who actually wants to uncover the AI through adversarial cross-examination.

The chat excerpt giving of 'Eugene Goostman' in https://en.wikipedia.org/wiki/Eugene_Goostman also suggests that it would fall apart immediately in an adversarial setting with the full three participants.

However, I do agree that we have made progress and that today's LLMs could hold up a lot longer in this harsher setting than anything we had before. Especially if you fine-tuned them properly to remove telltale signs like their inability to swear or their constant politeness.

> But to think we came up with good tests at the same time we invented electronic computers, should give you surprise.

I never claimed the Turing test is the best test ever, nor even that it's particularly good. I was saying that it hasn't been beaten in its original form.

For example, the Turing test isn't really a fine grained benchmark that lets you measure and compare model performance two multiple decimal places. Nor was it any good as a guideline for how to improve our approaches.

https://en.wikipedia.org/wiki/Loebner_Prize

Thanks, that seems to be reasonably close to Turing's rules. And predictably: no program ever convinced the judges that it was human and the real human competitor was a computer. No program passed the Turing test.

> In addition, there were two one-time-only prizes that have never been awarded. $25,000 is offered for the first program that judges cannot distinguish from a real human and which can convince judges that the human is the computer program. $100,000 is the reward for the first program that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. The competition was planned to end after the achievement of this prize.

> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?

In addition to the detectability problem, I wrote in the adjacent comment, this question can be further refined.

A Turing machine is an abstract concept. Do we need to take into account material/organizational properties of its physical realization? Do we need to take into account computational complexity properties of its physical realization?

Quantum mechanics without Penrose's Orch OR is Turing computable, but its runtime on classical hardware is exponential in, roughly, the number of interacting particles. So, theoretically, we can simulate all there is to simulate about a given person.

But to get the initial state of the simulation we need to either measure the person's quantum state (thus losing some information) or teleport his/her quantum state into a quantum computer (the no-cloning theorem doesn't allow to copy it). The quantum computer in this case is a physical realization of an abstract Turing machine, but we can't know its initial state.

The quantum computer will simulate everything there are to simulate, but the interaction of a physical human with the initial state of the Universe via photons of the cosmic microwave background. Which may deprive the simulated one of "free will" (see "The Ghost in the Quantum Turing Machine" by Scott Aaronson). Or maybe we can simulate those photons too, I'm not sure about it.

Does all of it have anything to do with consciousness? Yeah, those are interesting questions.

Yeah I think that's a good point, and my issue with Penrose. His point sounds more profound but it is like dropping everything in complexity analysis (Big O). An approximation only goes so far, and sometimes the specifics matter. More importantly, when they matter, they usually matter A LOT. I think people underestimate how many things we approximate, and approximate with only first or second order terms. But last I checked, most things are not simple harmonic oscillators (springs)[0]

[0] See the Taylor Series expansion. This is an only physics joke

Penrose is a dualist, he does not believe that function can be computed in our physical universe. He believes the mind comes from another realm and "pilots" us through quantum phenomenons in the brain.

Interesting. Does that fit with the simulation hypothesis? That the world's physics are simulated on one computer, but us characters are simulated on different machines, with some latency involved?

Its all pop pseudoscience. Things exist. Anything that exists has an identity. Physics exists and other things (simulations, computing, etc.) that exist are subject to those physics. To say that it happens the other way around is poor logic and/or lacks falsifiability.

eh, its possible that weird shit is happening in physics. however, there is no evidence that this is the case. its just vibes, really.

>Things exist

oh, cogito existo sum! checkmate theists!

dude, the simulation hypothesis does not mean things don't exist, it means they don't necessarily exist in the way you have, rather unimaginatively, imagined, and you have no way to tell.

and Occam's Razor does not solve the problem.

> … and you have no way to tell. This is exactly my point. If we have no way to tell, what experiment could you possibly use to test whether we’re in a simulation or not? The simulation hypothesis lacks falsifiability and is pseudoscience.

the "it's not a simulation" hypothesis lacks falsifiability.

also, falsifiability lacks falsifiability.

don't make up complex axioms and then believe in them on faith.

Occam's razor: if the universe can be simulated, then whatever form the simulation takes is the simpler explanation for the turtles all the way down.

if the universe can't be simulated, explain how you know that.

Which is—to use the latest philosophy lingo—dumb. To be fair to Penrose, the “Gödel’s theory about formal systems proves that souls exist” is an extremely common take; anyone following LLM discussions has likely seen it rediscovered at least once or twice.

To pull from the relevant part of Hofstadter’s incredible I am a Strange Loop (a book also happens to more rigorously invoke Gödel for cognitive science):

  And this is our central quandary. Either we believe in a nonmaterial soul that lives outside the laws of physics, which amounts to a nonscientific belief in magic, or we reject that idea, in which case the eternally beckoning question "What could ever make a mere physical pattern be me?”
  After all, a phrase like "physical system" or "physical substrate" brings to mind for most people… an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, it is *necessarily* so, as they see it. This is the kind of single-level intuition so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together.

Highly recommend it for anyone who liked Gödel, Escher, Bach, but wants more explicit scientific theses! He basically wrote it to clarify the more artsy/rhetorical points made in the former book.

It feels really weird to say that Roger Penrose is being dumb.

It's accurate. But it feels really weird.

It's not uncommon for great scientists to be totally out of their depth even in nearby fields, and not realize it. But this isn't the hard part of either computability or philosophy of mind.

We really ought to stop mythicizing people into superhuman heroes/geniuses.

Penrose is an authority in some fields of theoretical physics, but that doesn't give any value to what he has to say on consciousness or AI.

On that topic, he has clearly adopted an unscientific approach: he wants to believe the soul exists and is immaterial and seeks evidence for it.

hes a damn good mathematician. it is indeed weird to experience him not breaking down the exact points of assumption he makes on arriving at his conclusion. he is old though, so...

https://en.m.wikipedia.org/wiki/Nobel_disease

No, Penrose is not dumb. He gives a very good argument in his books on limitations of AI, which is almost always misrepresented including in most of this thread. It is worth reading "Shadows of the Mind".

I could be wrong but I don't think he's a dualist. He rejects human consciousness being computable on classical computers, like Turing machines, but thinks that quantum stuff is necessary. Quantum brains/computers are or would be computing in our physical universe, making him a materialist not a dualist. His big thing is using this quantum stuff to free his own mind from the confines of Gödel's theorems (I think for egotistical reasons.)

Simultaneously, actual dualists flock to his theories because they think association with Penrose lends credibility to their religious stuff.

All of this is a fine thought experiment, but in practice there are physical limitations to digital processors that don’t seem to manifest in our brains (energy use in the ability to think vs running discrete commands)

It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.

If human-like cognition isn't possible on digital computers, it's certainly is on quantum ones. The Deutsch-Church-Turing principle shows that a quantum Turing machine can efficiently simulate any physically realizable computational process.

> certainly is on quantum ones

I don’t think that’s a certainty.

In 50% of possible worlds, it’s 100% certain!

Does that math still check out if math works differently in other worlds?

Probably!

In about 50% of the worlds, there’s 100% chance, would you say?

I think the complication here is that brains are probabilistic, which admits the possibility that they can’t be directly related to non probabilistic computability classes. I think there’s a paper I forget the name of that says quantum computers can decide the halting problem with some probability (which makes sense because you could always just flip a coin and decide it with some probability) - maybe brains are similar

It is a big mistake to think that most computability theory applies to AI, including Gödel’s Theorem. People start off wrong by talking about AI “algorithms.” The term applies more correctly to concepts like gradient descent. But the inferences of the resulting neural nets is not an algorithm. It is not a defined sequence of operations that produces a defined result. It is better described as a heuristic, a procedure that approximates a correct result but provides no mathematical guarantees.

Other aspects of ANN that show that Gödel doesn’t apply is that they are not formal systems. Formal system is a collection of defined operations. The building blocks of ANN could perhaps be built into a formal system. Petri nets have been demonstrated to be computationally equivalent to Turing machines. But this is really an indictment on the implementation. It’s the same as using your PC, implementing a formal system like its instruction set to run a heuristic computation. Formal system can implement informal systems.

I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.

But surely any limits on formal systems apply to informal systems? By this, I am more or less suggesting that formal systems are the best we can do, the best possible representations of knowledge, computability, etc., and that informal systems cannot be "better" (a loaded term herein, for sure) than formal systems.

So if Gödel tells us that either formal systems will be consistent and make statements they cannot prove XOR be inconsistent and therefore unreliable, at least to some degree, then surely informal systems will, at best, be the same, and, at worst, be much worse?

Yeah I don’t know why GP would think computability theory doesn’t apply to AI. Is there a single example of a problem that isn’t computable by a Turing machine that can be computed by AI?

It does apply to AI in terms of the computers we compute neural networks on may be equivalent to Turning machines but the ANN networks are not. If you did reduce the ANN down to a formal system, you will likely find that in terms of Godels theorem that it would be sufficiently powerful to prove a falsehood. Thus not meeting the consistency property we would like in a system used to prove things.

I suspect that if formal systems were unequivocally “better” than formal systems our brains would be formal systems.

The desirable property of formal systems is that the results they produce are proven in a way that can be independently verified. Many informal systems can produce correct results to problems without a known, efficient algorithmic solution. Lots of scheduling and packing problems are NP-complete but that doesn’t stop us from delivering heuristic based solutions that work good enough.

Edit: I should probably add that I’m pretty rusty on this. Godels theorem tells ua that if a formal system is consistent, it will be incomplete. That is, there will be true statements that cannot be proven in the system. If the system is complete, that is, all true/false statements can be proven, then the system will be incomplete. That is you can prove contradictory things in the system.

AI we have now isn’t really either of these. It’s not working to derive truth and falsehood from axioms and a rule system. It’s just approximating the most likely answers that match its training data.

All of this has almost no relation to the questions we’re interested in like how intelligent can AI be or can it attain consciousness. I don’t even know that we have definitions for these concepts suitable for beginning a scientific inquiry.

Asuuming said formal system incorporates Peano arithmetic.

Heuristics implemented within a formal system are still bound by the limitations of the system.

Physicists like to use mathematics for modeling the reality. If our current understanding of physics is fundamentally correct, everything that can possibly exist is functionally equivalent to a formal system. To escape that, you would need some really weird new physics. Which would also have to be really inconvenient new physics, because it could not be modeled with our current mathematics or simulated with our current computers.

To be fair, I muddled concepts of formal/informal systems versus completeness and consistency. I think if you start from an assumption that ANN is a formal system(not a given), you must conclude that they are necessarily inconsistent. The AI we have now hallucinates way too much to conclude any truth derived from its “reasoning.”

> But the inferences of the resulting neural nets is not an algorithm.

Incorrect.

The comment above confuses some concepts.

Perhaps this will help: consider a PRNG implemented in software. It is an algorithm. The question of the utility of a PRNG (or any algorithm) is a separate thing.

On one level, yes you’re right. Computing weights and propagating values through an ANN is well defined and very algorithmic.

On the level where the learning is done and knowledge is represented in these networks there is no evidence anyone really understands how it works.

I suspect maybe at that level you can think of it as an algorithm with unreliable outputs. I don’t know what that idea gains over thinking it’s not algorithmic and just a heuristic approximation.

"Heuristic" and "algorithmic" are not antipodes. A heuristic is a category of algorithm, specifically one that returns an approximate or probabilistic result. An example of a widely recognized algorithm that is also a heuristic is the Miller-Rabin primality test.

https://xlinux.nist.gov/dads/HTML/heuristic.html

https://xlinux.nist.gov/dads/HTML/millerRabin.html

https://en.wikipedia.org/wiki/Miller%E2%80%93Rabin_primality...

“Algorithm” just means something which follows a series of steps (like a recipe). It absolutely does not require understanding and doesn’t require determinism or reliable outputs. I am sympathetic to the distinction that (I think) you’re trying to make but ANNs and inference are most certainly algorithms.

> On the level where the learning is done and knowledge is represented in these networks there is no evidence anyone really understands how it works.

It is hard to assess the comment above. Depending on what you mean, it is incorrect, inaccurate, and/or poorly framed.

The word “really” is a weasel word. It suggests there is some sort of threshold of understanding, but the threshold is not explained and is probably arbitrary. The problem with these kinds of statements is that they are very hard to pin down. They use a rhetorical technique that allows a person to move the goal posts repeatedly.

This line of discussion is well covered by critics of the word “emergence”.

This.

Heuristic or not, AI is still ultimately an algorithm (as another comment pointed out, heuristics are a subset of algorithms). AI cannot, to expand on your PRNG example, generate true random numbers; an example that, in my view, betrays the fundamental inability of an AI to "transcend" its underlying structure of pure algorithm.

1. If an outside-the-system observer cannot detect any flaws in what a RNG outputs, does the outsider have any basis for claiming a lack of randomness? Practically speaking, randomness is a matter of prediction based on what you know.

2. AI just means “non human” intelligence. An AI system (of course) can incorporate various sources of entropy, including sensors. This is already commonly done.

AI is most definitely an algorithm. It runs on a computer, what else could it be? Humans didn't create the algorithm directly, but it certainly exists within the machine. The computer takes an input, does a series of computing operations on it, and spits out a result. That is an algorithm.

As for humans, there is no way you can look at the behavior of a human and know for certain it is not a Turing machine. With a large enough machine, you could simulate any behavior you want, even behavior that would look, on first observation, to not be coming from a Turing machine; this is a form of the halting problem. Any observation you make that makes you believe it is NOT coming from a Turing machine could be programmed to be the output of the Turing machine.

> With a large enough machine, you could simulate any behavior you want

This is not exactly true, depending on what you mean by behavior. There are mathematical functions we know for a fact are not computable by a Turing machine, no matter how large. So a system that "behaves" like those functions couldn't be simulated by a TM. However, it's unclear whether such a system actually could exist in physical reality - which gets right back to the discussion of whether thinking is beyond Turing completeness or not.

> But the inferences of the resulting neural nets is not an algorithm

It is a self-delimiting program. It is an algorithm in the most basic sense of the definition of “partial recursive function” (total in this case) and thus all known results of computability theory and algorithmic information theory apply.

> Formal system is a collection of defined operations

Not at all.

> I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.

We have zero evidence of this one way or another.

—

I’m looking for loopholes around Gödel’s theorems just as much as everyone else is, but this isn’t it.

Excuse me, what are you talking about? You think there is any of computability that doesn't apply to AI? With all respect and I do not intend this in a mean way but just intend to rightly call all of this as exactly nonsense. I think there is a fundamental misunderstanding of computational theory and Turing machines, Church-Turing thesis, etc. any standard text on the subject should clear this up.

“Organisms are Algorithms.” --Yuval Noah Harari

>Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.

"Thinked information" is a colour not an inherent property of information. The fact that information has been thought is like the fact it is copyrighted. It is not something inherent to the information, but a property of its history.

https://ansuz.sooke.bc.ca/entry/23

> If we assume that that function is computable in the Turing sense

This is a big assumption. I'm not saying it's wrong, but I am saying it's not reasonable to just handwave and assume that it's right.

> Now, the question is: can we create a smaller function capable of performing the same feat?

Where does this question come from? Especially where does the 'smaller' requirement come from?

Gödel’s incompleteness theorem, and, say, the halting problem seem to fall squarely into the bucket of “basic computability theory” in precisely the way that “we think, that is a fact”, does not (D.A. hat tip)

You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers.

Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.

he starts with "consciousnes is not computable". You can not just ignore it as a central argument withouth explaining why your preference to think on it as basic computability theory makes more sence than his.

What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false

>> What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false

No this isn't the checkmate you think it is. It could still be a mathematical function. But every person transforming information into "thinked information" could have a different implementation of this function. Which would be expected as no person is made of the same code (DNA).

Not sure there aren't 10 other "lines of discussion/disagreement" on this, but the one I think might be most salient is Davidson's https://en.wikipedia.org/wiki/Anomalous_monism (Davidson was a student of Quine who people here might know more from his work in logic/kitchy programs-that-print-themselves games than as the bigtime analytic philosophy Word & Object, naturalized epistemology type work).

Note that using words like "function" and "mathematical" are more the biases of computer science/Penrose while philosophy of the mind & psychology has more typically used slightly different ideas of "attitudes" and "events" to guide the discussion. I don't think this really radically shifts many central disputes, except (which Penrose might view as a critical "except") for all the funny business quantum mechanics / quantum computation can potentially bring in and the (undisputed) physicality of our brains and (also undisputed) lack of understanding of many details of biological computation.

You're arguing that submarines swim.

No, I mean, it's nice but I don't think any of that works. You say "Therefore, there is a function capable ..." that is a non-sequitur. But, let's set that aside, I think the key point here is about Turing machines and computability. Do you really think your mind and thought-process is a Turing machine? How many watts of power did it take to write your comment? I think it is an absolute certainty that human intelligence is not like a Turing machine at all. Do you find it much more troublesome to think about continuous problems or is ironically more troublesome to discretize continuous problems in order to work with them?

I think it's much more difficult to fix a broken down human than a broken down car. Does it mean, by your logic, we are closer to cars?

What makes it an "absolute certainty" for you?

FWIW human brain does indeed consume a lot of energy, accounting for over 20% of our body metabolism. We don't know how to attribute specific watts consumed to specific thoughts because we don't understand the functioning of the brain enough, but there's no obvious reason why it shouldn't be possible.

Not every fact is computable. We are not Turing machines.

We don't know every fact, either, so I don't know how you can use that idea to say that we're not Turing machines. Apart, of course, from the trivial fact that we are far more limited than a Turing machine...

With sufficient compute capacity, a complete physical simulation of a human should be possible. This means that, even though we are fallible, there is nothing that we do which can't be simulated on a Turing machine.

May still only yield a philosophical zombie. You can simulate gravity but never move something with its simulation.

If I have a 9-DOF sensor in meatspace and am feeding that to a "simulation" that helps a PID coalesce faster then my simulation can move something. When I tell my computer to simulate blackbody radiation...

What you said sounds good, but I don't think it's philosophically robust.

I think you misunderstood my point. A simulation is never the actual simulated phenomenon. When you understand consciousness as a "physical" phenomenon (e.g. as in most forms of panprotopsychism), believing in being able to create it by computation is like believing in being able to generate gravity by computation.

we're out of my wheelhouse but it feels to me that entropy and gravity are fundamentally linked and a quick search shows i'm not alone: https://en.wikipedia.org/wiki/Entropic_gravity

read this as: literally creating gravity by simulating it hard enough.

I don't see how computation itself can be a plausible cause here. The physical representation of the computation might be the cause, but the computation itself is not substrate-independent in its possible effect. That is again my point.

I'm arguing with an AI about this too, because my firm belief is that the act of changing a 1 to a 0 in a computer must radiate heat - a 1 is a voltage, it's not an abstract "idea", so that "power" has to go somewhere. It radiates out.

I'm not really arguing with you, i just think if i simulate entropy (entropic processes, "CSRNG", whatever) on my computer ...

I agree and the radiation/physical effect is in my opinion the only possibility a normal computer may somehow be able to cause some kind of consciousness.

wow, we were at an impasse, but somehow i managed to make myself understood. Happy Mardi Gras!

> A simulation is never the actual simulated phenomenon.

That's not obviously true, especially given how the more we keep digging into physics, the more everything seems to be "just information".

Seems to me like wishful thinking. This would require an interface to connect to and there we are most probably in the physical realm again (what we can perceive as such).

To be clear, the topic of philosophical zombies has to do with consciousness.

You can simulate a black hole on a mainframe, but that doesn't mean the math is going to escape and eat your solar system.

We don't know if consciousness is computable, because we don't know what consciousness is.

There are suggestions it isn't even local, never mind Turing-computable.

the heat radiated is entropy and will eventually wind up in a black hole. we missed a step.

The entire p-zombie concept presumes dualism (or something close enough to it that I'm happy to lump it all into the generic category of "requires woo"). Gravity having an effect on something is measurable and provable, whereas qualia are not.

No, panprotopsychism and panpsychism are also very hot contenders. Considering the lack of any examples where computation itself yields measurable effects on reality why should it be the case for consciousness? It's wishful thinking of computer enthusiasts if you ask me.

Why should a complete simulation be possible? in fact there are plenty of things we can do that can't be simulated on a Turing machine. just one example the Busy Beaver Problem is an uncountable problem for large N, so by definiton is not coumptable and yet humans can prove properties like "BB(n) grows faster than any computable function"

Proving properties and computing values are quite different things, and proofs can absolutely be done on Turing machines, e.g. with proof assistants like Lean.

well you try feeding the Busy Beaver Problem with large N to lean then and see what comes out.

Do you think a machine proof of "BB(n) grows faster than any computable function" would require that?

no, see the problem is that the machine needs a well defined problem, and the "BB(n) grows faster than any defined problem" is well defined but you would not come up with an insight like that by executing the BB(n) function. that insight requires a leap out of the problem into a new area and then sure after it is defined as a new problem you enter again in the computability realm in a different dimension. But if the machine tries to come up with insight like that by executing the BB(n) function it will get stuck in infinite loops.

As long as you take the assumption that the universe is finite, follows a fixed set of laws, and is completely deterministic, then I think it follows (if not perfectly, then at least to a first order) that anything within the universe could be simulated using a theoretical computer, and you could also simulate a smaller universe on a real computer, although a real computer that simulated something of this complexity would be extremely hard to engineer.

It's not entirely clear, though, that the universe is deterministic- our best experiments suggest there is some remaining and relevant nondeterminism.

Turing machines, Goedel incompleteness, Busy Beaver Functions, and (probably) NP problems don't have any relevance to simulating complex phenomena or hard problems in biology.

Reply on news.ycombinator.com