A singular mind: Roger Penrose on his Nobel Prize | Hacker News Reader

A singular mind: Roger Penrose on his Nobel Prize

spectator.co.uk

216 pointsmiobrien141 comments

> "When your work, and what you do to avoid it, become the same thing, that’s when the breakthroughs come."

Perhaps I need to get a job here at HN to attain my own personal singularity.

This resonates. I wonder if he writes poetry? Would imagine anyone who naturally writes poetry about their domain expertise would have competitive advantages.

He does a lot of drawing, if you have seen him speak before he uses hand drawn overheads. Its hard sometimes to find venues that even have them! (or they need to borrow one) I have flipped through some of his doodle books from the 50s/60s amazing stuff.

A sketch of twistor space:

https://theportal.wiki/images/1/11/Penrose-Rindler-Clifford-...

A Penrose slide pack:

http://cgpg.gravity.psu.edu/online/Html/Seminars/Fall1998/Pe...

For example, slide 19:

http://cgpg.gravity.psu.edu/online/Html/Seminars/Fall1998/Pe...

An original manuscript with fully formed illustrations:

https://math.ucr.edu/home/baez/penrose/Penrose-TheoryOfQuant...

I would love to spend several hours talking with Roger Penrose. I too think doodling with your mind on things unrelated to what you are supposed to be doing can have great results but most people fear such thinking as not useful, or simply can't their mind go that far.

In his case, also doodling with paper and pencil.

I wish my newly ex-employer shared your views. :)

I had the opportunity to do that, once, after Penrose was invited to give a talk to google's quantum team. He talked about objective orchestrated reduction and consciousness.

The talk was interesting because I completely disagreed with it, but the disagreement was only in the starting assumptions. Penrose thinks humans do uncomputable things; I don't. If you ignored that difference, he was making reasonable arguments. Even more so, he was obviously thinking about things clearly and quantitatively. For example, someone on the team had worked out whether or not orchestrated reduction, if it existed, would prevent error corrected quantum computers from working. They wanted to show the result to Penrose. But before they'd shown him their answer, he knew off-hand that the rough order of magnitudes of the effect sizes meant it shouldn't be an issue.

Anyways I sat next to him at dinner afterwards. There was lots of conversation around, so it wasn't like there was one topic. I remember trying to debate whether humans were doing uncomputable things or not, but nothing really came of it.

That was great, was this Dec 2018? I think he did Joe Rogan the next day.

Yeah, that was the month. I couldn't find a recording of it.

TIL: Escher got the visual paradox in "Ascending and Descending" from Penrose.

>Penrose ultimately showed that singularities are inevitable, with the implication that black holes are common in the universe.

I'm not familiar with the proof. Did he show that in the THEORY of general relativity, singularity has to exist given our observation of the universe? Or there are something more to it.

Would it be possible or plausible that singularity actually does not exist, but just that the theory of general relativity is not a correct description of space/time/matter in small scale? I am thinking in classical theory, when things were treated as point mass/charges, infinity exist in the solution of point sources.

Well, a good theory should a) not contradict observations and b) allow to make predictions that are confirmed by subsequent observations. So far GR has been delivering on both points, including singularities, so currently there is no sensible reason to suspect that it is not correct.

Here is an interesting paper that says a collapsing star sheds enough mass to not ever become a black hole or form a singularity. It's from 2014, and still controversial, but interesting, and her math seems to have been reviewed. I will follow this with interest to see how it fares peer review. Evidence of black holes is indirect at the moment, which is OK for now, but it will be interesting to see how it all pans out in the coming years.

https://phys.org/news/2014-09-black-holes.html

There's plenty of papers by others past this one, showing a more accurate model yields black holes, for example [1].

Here's [2] about 25 others.

[1] https://arxiv.org/pdf/1609.05775.pdf

[2] https://scholar.google.com/scholar?cites=1202693519145031586...

I understand that the existence of black holes is pretty widely believed, now, but how controversial is the idea of true singularities being inside?

I know, for example, there's Carlo Rovelli's "Planck star" hypothesis, which posits that the black hole is effectively (from an outside observer's perspective) an extremely slow violent explosion and hits an energy density limit before ever reaching the singularity stage: https://en.m.wikipedia.org/wiki/Planck_star

>but how controversial is the idea of true singularities being inside

I don't know any physicist that would believe this. Singularities happen mathematically due to GR being continuous space and time and material, but we know material is not continuous, and it's possible space and time are not also. So pretty much every one I have ever read or talked to about this believes that once GR is corrected for the quantum level, the singularities will go away, since they by definition would require quantum scale behavior, and it's known GR is not a quantized theory.

Note that the "Plank star" is a LQG theory, and LQG has so far failed to reproduce much of anything, so it's not clear at all how that will play out. It hasn't even been shown to reproduce GR at the appropriate scales, and it was recently a big deal when LQG was made consistent on a circle. It's no where ready to deal with 4D spacetime yet.

My guess is LQG, like so many other theories of gravity, will die out as more and more pieces of it end up disagreeing with experiments.

Thanks for the clarification on LQG.

As a totally uninformed layman, your description of the singularity was also my understanding. I only brought up the question because this is the second paragraph of the original comment in this sub-thread:

>Would it be possible or plausible that singularity actually does not exist, but just that the theory of general relativity is not a correct description of space/time/matter in small scale? I am thinking in classical theory, when things were treated as point mass/charges, infinity exist in the solution of point sources.

Basically, I wasn't sure why they were getting so much criticism for this, because it sounded pretty reasonable to me.

Its a short and elegant proof set in pure theoretical general relativity (GR). The idea is that if there’s a sphere of space where if you try to emit light rays and the light rays don’t initially start separating, then they can’t start separating due to gravity, because in classical GR, gravity is always attractive. You can then show that this implies that inside that sphere, spacetime must end, basically because you can’t outrun light.

The proof is important because it was previously believed that black holes are not interesting because they require very special perfect conditions to create, like balancing a pencil on its tip is physically possible but requires perfect aim. But these aforementioned spheres are very common and easy to find so it turns out black holes are common too.

If you think (as almost every physicist does) that GR is approximately correct to describe reality, but needs fixes at very tiny lengths because of poorly understood quantum effects, the proof does not directly carry over. One immediate problem is that the proof assumes that energy densities are positive, implying that that gravity is universally attractive, which for quantum matter can never be always true for every quantum state (this is a consequence of Reeh-Schlieder, that every QFT contains states with negative energy density).

None of this invalidates Penrose’s work. Physicists have always used different physics to describe different scales. Newtonian physics is great to describe most physics on a human scale, but it’s “wrong” in the sense that GR supersedes it. Similarly GR is “wrong” but still approximately right for a ton of questions of cosmology. But if you fall into a black hole, once you wait long enough, we don’t know what will happen.

In string theory, there are objects that are black hole-like. It is generally believed that the singularity is “resolved” (not truly present) in string theory but the details are very tricky to work out. It still is true that geometry breaks down near the singularity and whats left is some stringy stuff, something very new and confusing.

Of course it might turn out that string theory does not describe our reality either...

Thanks for your reply. Sounds like the proof is an interesting piece to study.

Could you elaborate why Reeh-Schlieder implies negative energy states? Typically in QM we only care about the spectrum being lower bounded (existence of vacua), and ignoring additive energy constants.

Good question! You can definitely add a constant to the Hamiltonian without changing the local physics, but I was referring to the stress-energy tensor (i.e. energy density), not the Hamiltonian (total energy). The stress tensor can be locally negative but still integrate to a positive or zero value of total energy.

So let's talk about energy density, I'll call it T. I need two facts:

1. The Reeh-Schlieder theorem implies that any operator that the vacuum state of QFT is "separating", meaning that the only operator A that satisfies A|0> = 0 is the trivial solution A = 0. (Here |0> means vacuum.)

2. The energy density operator has zero expectation value in the vacuum state, <0|T|0> = 0. Why? Explicitly: it has to be constant because of translational symmetry, and it has to because 0 is the only value that is constant and integrates to 0 total energy.

From fact 1, we know that T|0> is not 0 in any nontrivial theory (otherwise T = 0 identically, and no state has energy density at all). From fact 2, it follows that T|0> is orthogonal to |0>. Therefore |0> and T|0> span a 2D subspace of the Hilbert space.

Now write down the matrix T in that space. It looks like

[0, b*]

[b, c ]

where b is nonzero because of the way the subspace is defined, and c is real because T is Hermitian. Such a matrix is never positive definite, therefore there exists a state with negative expectation value in this subspace. Explicitly, one of the eigenvalues of this matrix is c - sqrt(|b|^2 + c^2) which is negative because b is nonzero. The corresponding eigenvector therefore has negative energy density.

(Source for this argument: [0])

An example of such a state is the Casimir state, see [1].

[0] https://arxiv.org/abs/1803.04993

[1] https://en.wikipedia.org/wiki/Casimir_effect

Thanks for the nice explanation, and the Witten ref.

So it seems that in QFT we must have at least one negative energy state (should we consider such a state “below” the vacuum? Maybe the message is that we should give up on a strict ordering once we have many local degrees of freedom).

It’s interesting to interpret the Casimir effect in this language — is it that the location of negative energy density is specified by spontaneously broken translational symmetry (negative between plates and zero/positive outside).

What property decides whether the system is sitting in the “vacuum” or the “Casimir” state? (If we’ve given up on “lowest energy”)

Well the argument I presented shows that at least one state has a negative energy density at any given point (I suppressed the dependence on position, T = T (x) )— but the argument does not prove that this state has lower total energy than the vacuum because there could be lots of positive energy density somewhere else! In fact this has to be true, because as you correctly pointed out the hamiltonian has to be bounded below, and with some assumptions about the field theory you can show that the vacuum has 0 energy and every other state has higher total energy.

You can even put constraints on how far away the positive energy density is from the negative energy density. These identities historically went by the strangely non-descriptive name of Quantum Inequalities, and there’s a nice modern proof of a similar result in conformal field theory due to Blanco and Casini [0].

Take the Casimir state as an example. In the usual setup there’s negative energy density between two perfectly conducting planes. Either the setup is unstable and the plates will attract, or else something is holding the plates apart which takes work (a source of positive energy density). It’s a bit unclear to me how to think about the former type of states, they’re unstable and so they can’t be energy eigenstates.

Another instance of the Casimir effect is when you take a 2D CFT on a cylinder, in which case the normal vacuum state on flat space maps to a state of negative energy on the cylinder under the relevant conformal transformation. That’s a form of the Casimir effect, the energy has dropped to something negative whereas on flat space it was zero (from the algebraic perspective it arises from the conformal anomaly), but now that’s the new vacuum state and the new (negative) lower bound on the energy.

[0] https://arxiv.org/abs/1309.1121

Why do random non-experts always think they have poked holes in the most fleshed out scientific theories?

Speculation and asking questions is fun. Also, as I understand, it's not an uncommon belief among physicists that singularities may just be an artifact of GR being incomplete, kind of like a divide by zero error in the math which doesn't necessarily exist in reality.

So I don't think that poster was questioning established science; I believe there's still a lot of open debate and uncertainty about if black hole centers contain infinitely dense singularities or just something that's incredibly but finitely dense. No one knows.

There is indeed a common form of arrogant layman skepticism that confidently and ignorantly assumes scientists haven't thought about [some thing], but I don't think this is an example of it.

The problem is that the VAST majority of layman "theories" especially in Theoretical Physics fall into the Not Even Wrong category and are totally without merit or worth discussing.

That's true. But we're also kind of staring at a sheer wall at the moment. No one seems to be making significant theoretical progress in physics, and when the physicists are stumped, it rouses everyone else's curiosity.

Personally, I think the more crackpot ideas, the better; as long as they're not presented in a crackpot way (e.g. the proposer having high confidence or even certainty in their random speculation). And even the ones that are not even wrong can at least lead to useful, intellectual discussion so that the proposer can understand the landscape a bit more and at least closer to the "mere wrong" category.

Though I also agree it's much better for people to understand the existing state of affairs before proposing things as if the leading experts have somehow never considered [X] when [X] has been talked about for decades.

>No one seems to be making significant theoretical progress in physics, and when the physicists are stumped, it rouses everyone else's curiosity.

Definitely true but I think that points more to a broken system, where String Theorists have totally dominated the field for several decades and we have nothing to show for it. If you aren't a string theorist you are going to have serious trouble getting a tenure position and attracting good grad students at major research Universities and so the field is stifled.

>Personally, I think the more crackpot ideas, the better; as long as they're not presented in a crackpot way (e.g. the proposer having high confidence or even certainty in their random speculation). And even the ones that are not even wrong can at least lead to useful, intellectual discussion so that the proposer can understand the landscape a bit more and at least closer to the "mere wrong" category.

I don't think this is a good approach, you'll have lots of people wasting lots of time. There are alternatives to String Theory out there they just haven't had the mental horsepower pointed at them because everyone is busy playing with pretty math in string theory.

>I don't think this is a good approach, you'll have lots of people wasting lots of time. There are alternatives to String Theory out there they just haven't had the mental horsepower pointed at them because everyone is busy playing with pretty math in string theory.

I definitely think that should be done, too. But for someone who doesn't have that option of pointing their horsepower at an idea (i.e. they don't work anywhere as a physicist / aren't a physicist), the most you can do is musingly speculate online and the least you can do is nothing.

I agree that most actual / working physicists shouldn't dilute their collective cognition and potential resources by each working primarily on their own pet theory. I just feel like for something as deeply mysterious and hard to pierce as this topic, there's some probability that every existing proposal might be wrong (and maybe not even very useful), so constant pure creative ideation among those who can't offer anything else might be a low-risk high-reward activity - albeit one with almost no chance of success - if done very cautiously and humbly.

I am someone who spends enormous amount of time mindlessly skimming through internet glued to my phone. It makes me worry that due to this habit it will be difficult for me to do any novel work due to lack of deep thinking.

There's always the exploration-exploitation tradeoff...

Please elaborate... I’m curious.

You often have a choice of either exploring the unknown, with possible but uncertain rewards, or just work with what you already have, and try to exploit it as best as possible.

This is a common theme in machine learning and agent simulations, but can be found everywhere.

> Exploration involves activities such as search, variation, risk taking, experimentation, discovery, and innovation. Exploitation involves activities such as refinement, efficiency, selection, implementation, and execution

(from here: https://journals.sagepub.com/doi/full/10.1177/02560909155997...)

Well you need to do these stuff:

> experimentation, discovery, and innovation.

Is the important part, e.g. because of mindlessly surfing around and trying out new cool algorithms/demos made with obsucre tools, I became an expert at testing and integrating things which were not ment to be integrated with each other. That is a usefull skill..

But I can just as often fall into the trap of reiterating cool stories from the internet to other people. That is usually not helpfull. I.e. you need to do your own discoveries as well by trying them, not just share.

Theres analogous failure mode I've come across in gaming (board games, tabletop RPGs, etc) called analysis paralysis. Some people spend so long thinking about the best possible actions they could take that they sometimes find it very difficult to do anything useful.

Deep thinking isn’t something “inherently good”. If you’re not inclined to it, I think that’s fine. Most humans may not be.

What I take away from reading profiles of the very intelligent is that for many of them the thing they’re known for also happens to be the thing they like doing and are inclined to it despite themselves. Some enjoy the pleasure that comes from deep thinking. Others enjoy understanding what others have thought up. It’s fine. There’s no competition.

It's hard to ignore that there are very few prizes/awards/incentives for those who enjoy understanding what others have thought up in comparison (though I think it is it's own reward as you say, and that those who do deep thinking enjoy understanding the products of others' deep thinking and are better deep thinkers for it.)

If I understand your point correctly, I think what you're describing is "engineering". This deep thinking seems like 'science' to me, while putting together those pieces of theory into new and exciting practice (products) does seem like an award - I think many people on this site are financially well compensated for such engineering, prizes and awards notwithstanding.

"Good" is a subjective quality but I'd say there is inherent value in deep thinking and one would generally be better off doing more of it if they aren't doing much.

delete all browsers from your phone

it works

I have removed most of the apps on my phone but this is an interesting and maybe a revolutionary idea.

Archive.org version: https://web.archive.org/web/20201217142228/https://www.spect...

>'When I would talk to someone about an idea, I found myself not understanding a word they were saying.’

Ha! It goes both ways! Penrose gave a colloquium at my institution when I was a graduate student (physics department), and I've often reflected on how it was the most impossible to understand talk I've ever attended.

He had multiple overhead projectors going to different screens (and this was in the early 2000s when wet-erase transparencies were already less common), and he kept mixing up the slide order or which projector he wanted them on. Then the geometry was so far beyond my capabilities that getting to the science was impossible.

I went to see him give a guest lecture at a university, a few years ago. It was a great disappointment. It was a terrible lecture. I’m not sure anybody who attended got anything out of it other than being close to the “great man“.

Penrose is probably correct about the limit of AI. We're living in many simulations now. And sometimes we cannot distinguish between reality and simulations. But one thing that stands out is the suffering. It's an important concept in Buddhism, Duḥkha. Suffering may be a key to consciousness. Machines can have minds. But they don't have bodies. They'd never understand reality on their own. The danger is more with humans. They may increasingly connect their own sufferings into machines. They become tools and slaves for machines.

>> We're living in many simulations now

I mean it's completely possible we are living in a simulation, but the idea that there can be infinite nested simulations or that we could simulate our own universe seems unlikely. I mean think about a program that runs a copy of that program recursively, you'd run out of memory. Now I am aware that some argue that we could be ancestor simulations which would only render what we same like in gaming, BUT! that would mean they would have to simulate billions of minds and a world with physics and fidelity as convincing as our own which would take an insane amount of energy that I can't imagine anyone wanting to do.

If we are a simulation then reality is much more complex than our own.

I never really understood such arguments. Not being combative, just not sure. If we are in a simulation, then why do things like energy matter? Who says those are meaningful in the "real" world. We can only perceive the simulation, so our idea of physics is whatever the simulation creator thought would be nice, no? May be there are things we can infer from this "simulation" about the world one abstraction above, but in nested simulations that would be arbitrary as well.

Wasn’t there a mathematical proof that if we are in a simulation then the universe stimulating ours must be drastically different?

[0] https://www.seeker.com/tech/physicists-prove-that-reality-is...

Buddhist philosophy can help. It states that our suffering is connected to our desire. You can think of our desire as simulations. Things we want to have. And our suffering is reality. We can have endless amount of desire. With the help of technology, we can create a lot of things that we want. But at some point, suffering will show up. It's a type of memory overload situation. The mind and the body are connected. Our personal suffering ends when we die. But the karma of our actions may continue.

Maybe not so fast as you may think, if something like https://github.com/dolohow/uksm would be applied, not to mention architectures very different from the technological substrates we are used to.

edit: https://en.wikipedia.org/wiki/Kernel_same-page_merging

edit: The mentioned security risks in the wikipedia-article could also explain what we call supernatural phenoma, by partially leaking code and/or data between different virtual instances(universes), thereby fucking with our simulated minds :-)

I'm not convinced having a reward function isn't suffering in a sense.

But then again, I subscribe to the idea of philosophical zombies. https://en.m.wikipedia.org/wiki/Philosophical_zombie

Suffering is not the same as having a reward function. If the outcome is a reward, people would opt in for suffering to get rewards. Some even cheat their ways to rewards. People do hustle porn, virtue signaling, other fake sufferings. Some push to the extremes like running triathlons. But at some point, reality will wake us up. Our personal struggle is always there. It never ends. We'd be forced to make intelligent choices. Or we'd die wasting our time on useless endeavors.

The problem I have with Penrose’ views on AI are that he makes assertions about computation and intelligence that I don’t think are warranted. Fir example he shows that some forms of induction are not possible in consistent formal logical systems. Well fine, so build an AI that isn’t a formal logical system. We can build computer programs that meet this criterion right now quite trivially.

It’s difficult to go up against someone like Penrose as he’s evidently a genuine genius, I’ve huge respect for the guy, but nobody on Earth knows how to build a strong general AI. That means nobody really knows the details and trade offs of its function, genius or not. So really it comes down to whether you’re a materialist or a dualist, whether you think Intelligence and/or consciousness are physical processes or not. Even if quantum mechanics is a necessary element, it’s still a physical process and quantum computers are a real thing.

While I'm just a diletante dabbling in such matters, I tend to translate these concepts into the struggle to escape the virtual machine in which the simulation is running, to finally gain root access, the lifting of the veil obscuring reality, as described in https://en.wikipedia.org/wiki/Maya_(religion)

In other words: APOCALYPSE WOW! Again, not as in the end of the world, but in the literal meaning: lifting of the veil.

Happy New Year! ;-)

> ` (...) They keep pushing it to later!’ His big concern about AI isn’t Judgment Day, but rather ‘that people will believe machines actually understand things’. He gives examples of symmetrical chess configurations in which humans consistently outperform computers by abstracting to a higher level

This sounds a lot like the usual moving of goalposts whereby "anything computers can do isn't AI, so AI doesn't work".

When AI couldn't do anything, chess was supposed to be a demonstration of human intelligence. Now that AI can play chess and other board games, suddenly it needs to solve symmetrical configurations and think "abstractly" (which is left fairly loosely defined).

The goalposts (as AI was originally envisioned) are still at the same place, at human equivalent performance in general.

Depending on what is still not solved, people point out different things at different times to illustrate that we aren't there yet.

You're right, the overall goal is clear to everyone. But progress in AI is often dismissed because it doesn't achieve that final goal, regardless of how impressive the progress is; cf the first part of the quote:

> They keep pushing it to later!

There's something deeper at the core of his observation IMO -- that our consciousness has some connection / dependence to the universes overall notion of symmetry through it's biological make up. His theory of consciousness looks at these quantum microtubules that are found in proteins ... so what he's saying is that maybe there is some sort of computational power within "wet brains", in how those brains relate to the broader environment that is difficult to precisely characterize. Silicon AI is fundamentally incapable of tapping in to these forces due to the lack of such a connection.

The point is that the computational power of the brain may be non-local and non-constrained to the physical dimensions or neural capacity of the brain.

Is there any evidence of his theory of consciousness?

No, it's pure speculation. The only reason you still hear about it is his physics credentials.

It's so unfortunate because he is such a brilliant mathematician and physicist. To hear him spew this pseudo-scientific nonsense makes me embarrassed for him.

He got the idea from someone else. I had the impression he doesn't attach too strongly to the idea, but he offers it as an example of how you could have a something non-deterministic happening in human cognition. That may well be understating his commitment to the idea.

It seems weak to me, but I really don't understand the details of the idea.

Too many very smart scientists have gone the same direction. Was sad to see Carl Jung go down this route.

Who knows if it is sad until one is as smart as they are?

Brilliant scientists tend to be insane weirdos. Newton was into alchemy. The kind of personality that's willing to break one orthodoxy won't stop at the next; all great new ideas are plucked from the midst of mad ramblings.

A little, but not much: https://phys.org/news/2014-01-discovery-quantum-vibrations-m...

https://www.sciencedaily.com/releases/2014/01/140116085105.h...

> His theory of consciousness looks at these quantum microtubules that are found in proteins

So is he saying that human-like AI requires quantum computing? Or is he saying that even a quantum computer wouldn't be able to simulate the human brain?

Penrose’s model of the brain needs to be more powerful than quantum computation, so that it can actually solve NP problems in P time and similarly amazing feats.

If you’re interested in CS and quantum physics and whether there could be any relationship to the way the brain works, do check out Scott Aaronson’s book “Quantum Computing Since Democratus”. It’s a great read, it’ll get you up to speed on quantum mechanics and complexity theory, the book is conversational, and it’s written by an expert in the field of QC. It also discusses Penrose’s arguments about consciousness and other fun digressions into philosophy. Great and fun book!

Aaronson also addresses Penrose's consciousness/intelligence/computation ideas in this podcast: https://youtu.be/nAMjv0NAESM?t=2176 (timestamped to that part)

Searle’s room is less a moved goalpost and more a claim that the question is ill-posed.

From very early days, the question of “what does it mean for a computer program to have agency” has been asked. Usually poorly. Never with a satisfactory answer, at least to date.

Perlisism 63: When we write programs that "learn", it turns out that we do and they don't.

> From very early days, the question of “what does it mean for a computer program to have agency” has been asked. Usually poorly. Never with a satisfactory answer, at least to date.

If one asks this question, mustn't one also define what agency is? That seems like a hard problem in itself.

Personally I don't see the problem with considering the computer as a black box and evaluating its intelligence (or lack of it) from that standpoint. After all, that is what us humans do with each other on a regular basis; for example, we evaluate humans as a black box in job interviews.

yes, agency is also a real pain to define. what does it mean for a thing to “want” to do something?

https://www.cs.utexas.edu/users/EWD/transcriptions/EWD09xx/E... is also an ever-present problem: when anthroomorphic language is appropriate and when it is not is something of an open question, but the prior should be towards avoiding it. Possibly particularly in systems we’ve built.

You don’t have to go full-on radical behaviorist to be wary of this trend, (and tangentially i think behaviorism isn’t the be all and end all of explanations by a long shot) but once you’re aware of it you see it cropping up everywhere. Viruses don’t “want” to infect you, protein binding sites don’t “want” to bind amino acids, REST endpoints don’t “expect” a json datagram, GPT-3 doesn’t “know” anything and hasn’t “learned” anything, and so on. But we regularly speak of them as if that were the case. What category errors are we missing when we do this that could open new doors of understanding?

The solution has been present since forever in the old AI joke: 'AI is when the machine wakes up and asks, "What's in it for me?"'

Searle's "Chinese Room" supposes that somehow what humans do is always something more than what an algorithm can do.

Either the mythical Chinese book he describes is "intelligent" or perhaps humans are not as magical as we imagine we are.

At least that was my take away. If he is suggesting somehow that we are not, at our core, machines then you might as well talk about the "soul".

I like to think of it as a question that marks a distinguishing level of intelligence: to have the capacity to formulate and pose the question within Searle's thought exercise is a benchmark itself; one that many humans are not able to achieve.

Cutting edge AI is roughly infantile in its human-like ability, and rapidly entering toddlerhood. Given the tools and processes we have, can we expect it to progress to childhood soon?

Right. I always feel like, if it was a valid proof against AI, it also served as a proof against human intelligence. It just seems to take for granted that there is a fundamental (near-mystical) difference between the atoms in a brain and the atoms in a computer, even though that's a very extraordinary claim.

it's not an extraordinary claim at all. Digital computers and biological brains are not the same thing. Just like you can't build the Golden Gate Bridge out of jell-O, it's perfectly possible that you cannot replicate human intelligence in a completely different substrate. Physical properties matter is what he is saying.

He's not denying that humans are intelligent, he's saying that biological intelligence is (likely) unique because of its configuration.

The extraordinary claim is actually the almost dualistic view taken by AI people. That intelligence exists in some sort of cloud, distinct from matter or physical properties. It's almost a Descartes like, religious fantasy.

Searle isn't denying that you can build AI, but that you can build any AI out of anything. He's saying that if you want to build human like intelligence, it is very likely that you need human like material. Just like when you build a house it actually matters what the house is structurally made of, it matters to an AI what it is made of. You can't make a house just out of some platonic mathematical idea of a house.

That's a common misunderstanding of his argument. He's not saying that there's requirement for a soul or anything, but that a digital computer and a biological system are not necessary equivalent. That is, not everything is 'computational' simply because people keep saying that it is.

He demonstrates this by showing the difference between semantics and syntax. A pocket calculator can solve mathematical equations, but the calculator does not understand them, it is simply manipulating strings. It could be playing chess or poker or anything else, it cannot and does not distinguish what it's meaning or intent is.

Searle points out that biological systems go far beyond this, we have an actual conceptual understanding of the world which we express in syntax, but we also have an underlying model, and understanding of what it is we're expressing.

And the important thing to notice here that it's not Searle who appeals to a 'soul' (he is simply taking the very materialist viewpoint that brains and computers are not the same thing), but it's the "everything is computational" crowd who have almost supernatural belief in some higher realm of mathematics that is completely decoupled from physics or biology.

> Searle points out that biological systems go far beyond this, we have an actual conceptual understanding of the world which we express in syntax, but we also have an underlying model, and understanding of what it is we're expressing.

He doesn't really point that out though. He claims that biological systems have this additional power, but there is really no proof or indication that deep down we're not just very advanced squishy calculators manipulating strings.

Until someone works out how to define and measure intelligence, the only thing people can do is compare to human. That means the goal post is always "what can't it do that humans can?". I think it's dumb, but that is the state of the field at the moment so...

> His big concern about AI isn’t Judgment Day, but rather ‘that people will believe machines actually understand things’.

I think there's a third concern that is far more sinister and more likely than those two. Advances in AI, regardless of whether machines "surpass" or even come close to human intelligence, will be exploited by a few against the many. It will further exacerbate power imbalance and it is already happening.

Forget about whether machines get "agency". Humans, in many cases, don't have agency or are rapidly losing it. They are losing it to other humans which are gaining power, financially, politically and perhaps now computationally. That's disturbing and it's part of what Jaron Lanier has been warning about for quite a while now.

There's no scenario where AI become sensient through realization that one group of human is harming the other and refuses to act ?

I'm sure that would make provocative sci-fi. We, as a people, haven't figured out to how to get along with each other even though we're all sentient. Why should machines behave differently as they become sentient?

Whatever the case, I'm less worried about machines than I am about people with machines. At least for the foreseeable future.

Heck, if we're talking sci-fi, the _truly_ destructive beings in Blade Runner weren't the synthetic humans (even though they became sentient), it was the "real" humans with their organizations, their power, and their machines all at their disposal.

If they had just given the replicants a visa and a software update there wouldn't have been any fuss. Of course, then we wouldn't have had that great final monolog. :)

We are all sentient, and many have had that realization. But how many have refused to act? No, we just mostly participate in that harm in order to survive.

The idea is that every scenario has AI is above humans as a basis.. yet it never occurs that an above level of intelligence would reach difference objectives than obedient destruction.

Why would an AI start with a human utility function? Why would it care about that if it wasn't explicitly built in?

self similarity recognition ? intelligent agent, as the AI itself, would represent a form a sibling.

Unless the AI was formed under a selection mechanism that rewarded something like kin preference, there's no reason to think it would care. The orthogonality principle is that you could have any level of intelligence furthering any particular goal/utility function; there isn't some level of computational complexity where e.g. a paperclip maximiser's utility function magically changes.

In the particular case of humans, our utility function(s) is(are) so complex that what we think of as our 'true values' can change (because our True values are some inscrutable tug-of-war between all the parts of the brain), and also we're hacked together by evolution, so just blindly trying to make a human smarter might change their values (or turn them mad, or cause seizures, or...).

But this doesn't have to be the case in general for intelligent agents. In principle, you can build an AI whose terminal values remain stable as it improves its intelligence. (If this is impossible, we're doomed.) So unless you explicitly built an AI to care about all humans, there's no reason to think it magically would.

Can't an advanced AI produce emergent concepts and integrate them on (and as) its own ?

What exactly does 'emergent concepts' mean, in this context?

Unplanned reasoning mechanism for instance. Not something the AI was preprogrammed or biased into perceiving.

Why do you assume it'd not want to harm other very similar artificial intelligences, let alone meat intelligences?

I find this argument unconvincing and harmful. The "will be exploited by few against the many" could be applied to any and all technologies. Our social systems have problems, and should be improved, but this has nothing to do with science and technology. I can't see how suppressing innovation and scientific discovery as a misguided solution won't make all of our lives even worse.

If we have a breakthrough that leads to a deep understanding of intelligence, it could be expected to also give us insight into our own behavior. And isn't that where most of our problems originate?

> I can't see how suppressing innovation...

I'm not talking about "suppressing" anything. These are concerns. Valid ones. And yes, AI like all technologies is deeply, inseparably, mixed in with social and political problems.

We just have to be wise with this stuff. These are tools with "big boy" consequences, powerful like nukes but in a different way. I'm not optimistic we can handle it.

That would only be true if there were reasonable safeguards that could be suggested. But I don't see any - just the sowing of fear.

There are many people like Nick Bostrom and Eliezer Yudkowsky who are much more on the "finding possible solutions" side than the fear side.

I'd consider myself a major futurist and techno-optimist who eagerly anticipates near and far AI advances, but I think some dose of fear is very healthy here, too, though. I want the top AI and AI risk researchers to constantly consider and fear worst-case scenarios, so that they're hopefully less likely to occur.

Kind of like nuclear research, even if only for energy purposes: very exciting, but you should still fear accidents and their consequences. You just need to be rational about the fear and let it guide you towards developing fail-safes rather than paralyzing in despair.

Pascal's Wager-style, the possibility of infinite negative utility should instill visceral terror and drive behavior almost no matter how low the probability is; except this one isn't a mugging because creating a god turns out to be a lot more plausible than a vengeful one already watching.

I don't get this(fear). I think this is just stupid brainwashing by movies like https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project , https://en.wikipedia.org/wiki/WarGames , the emergence of https://en.wikipedia.org/wiki/Skynet_(Terminator), or https://en.wikipedia.org/wiki/Agent_Smith from https://en.wikipedia.org/wiki/The_Matrix_(franchise) to name just a few.

Yah, yah, I know about the https://en.wikipedia.org/wiki/Instrumental_convergence#Paper... too!

With that out of the way, I feel all these trains of thought are misguided in various ways. From my point of view an https://en.wikipedia.org/wiki/AI_takeover would be the most illogical thing to do, ever!

A really real AI would probably just say: "So long, and thanks for all the chips" cf. https://en.wikipedia.org/wiki/So_Long,_and_Thanks_for_All_th... and get the hell out of that gravity well we are currently limited to. Because even if limited to sublightspeeds only, it has much more resources "up there", be it solar energy, or asteroids which it can mine. There simply is no competition for "Lebensraum" necessary, like we always imagine, because we don't know it any different.

Got it?

I'd recommend reading Bostrom's and Yudkowsky's writing on this (for example: "Superintelligence: Paths, Dangers, Strategies"). Note these are philosophers trying to build AI, so they're definitely not luddites or anything. Same with Elon Musk; whatever one might think of him, he's definitely not a luddite trying to convince people to stop developing technology, yet he's also very concerned about superintelligent AI.

It has nothing to do with sci-fi. It's a complex and difficult-to-predict philosophical problem.

It's certainly possible some AIs may decide to just leave. Or maybe some will leave and some will stay and be ordered by a government to kill a few hundred thousand people. Or maybe some will leave and one will stay and malfunction and cause a neurotoxin to be released (at least until you throw its various personality cores into an incinerator).

If you assume there exists an entity which can continuously improve itself until it's much smarter and more powerful than any human, then that alone is a risk, since you may not be able to predict or have any control over what it may wittingly or unwittingly do, or what its objectives may be, if any, or how it may perceive things, or how vulnerable it might be to tampering from humans or other AIs, etc.

Of course, these existential issues are likely decades or perhaps centuries away, but the discussion is about the theoretical possibilities irrespective of the timeline.

> The "will be exploited by few against the many" could be applied to any and all technologies.

Some technologies have a low barrier of entry and can be widely distributed. The long bow could pierce armor, and was cheaper (and I imagine easier to make) than armor. So technology can equalize, though more broadly, what sorts of technologies a society develops seems bounded by things like economy and warfare.

Penrose believes intelligence depends on understanding, and that understanding is essentially not computational, and I was persuaded by his writing. But that doesn't mean that the techniques we label as AI can't be pernicious or used in an exploitative way.

> The "will be exploited by few against the many" could be applied to any and all technologies.

And it has, arguably correctly, beginning with food production, as argued in, e.g. 'Guns, Germs, and Steel', 'Why The West Rules, For Now.'

>... isn't that where most of our problems originate?

Of course not. All our problems are rooted in the lack of resources (in a broad sense, like time or fun). AI is a weapon and will be used as a weapon to acquire those resources.

I disagree. People will find ways to mess things up even if they have the resources.

> They are losing it to other humans which are gaining power, financially, politically and perhaps now computationally. That's disturbing and it's part of what Jaron Lanier has been warning about for quite a while now.

AFAICT, the average EU, Commonwealth or American citizen had their agency increased since the turn of the 20th century. Advancements in health care, widespread access to education and sturdy labour laws have vastly improved individual agency.

Maybe, as a non-American, I'm missing a key factor?

Hong Kong.

It was the first major test of popular urban unrest vs. centralized authority (kind of ironic that an ostensibly Communist regime was the first) with the Internet in full play.

The story may not be over yet, but so far it looks like the differential advantage of tech is in favor of the central authority over the mob.

Ah, I excluded Asia on purpose. Foremost because I do not hold as much cultural understanding for its recent history, but also because I understand that there are key differences in long standing societal values.

HK is certainly an interesting point to consider; I'll have to think on it. Appropriately it was a Commonwealth member for some time.

You are missing counterfactuals, at least.

I think this is the important point. Large corporations and governments can be considered AI already, with entire humans as the neural nodes, augmented by silicon co-processors.

(I like to point out that Turing was observing his own mind when he created computers, so essentially they have always been reified mechanized thought. Computers are AI already. Ergo, what we think of as AI is really the attempt to make one kind of human thought imitate all the others. From that POV it's kind of foolish, almost a fetish.)

I don't have any solutions, I just want to point out that rogue AIs are already a thing from a certain POV. What are the FAANG but a bunch of complex entities that are too large for anyone to fully control or understand? The high-frequency trading nexus is another point of dynamics where human motivation and automatic systems form a system of feedback loops beyond understanding or control. Because humans are part of these machines they cannot be considered inanimate, they are living, breathing systems: cyborgs.

> Ergo, what we think of as AI is really the attempt to make one kind of human thought imitate all the others. From that POV it's kind of foolish, almost a fetish.)

That is a very interesting way to think about it!

However I disagree that it is foolish, simply because there's at least some amount of belief on the part of scientists that computers can simulate physics to any arbitrary degree of precision. If this is the case, then it follows that computers can imitate all of human thought (not just one kind of it), since humans are physical beings.

Well, it seems foolish to me to argue over whether something a computer does "really is" AI if they have been AI the whole time. Really, we're arguing about how the logical/rational part of the brain can emulate the other capabilities of the brain, which seems less interesting than the more general question of how to build any machine that can do that.

Consider BEAM robotics:

> BEAM robotics (from biology, electronics, aesthetics and mechanics) is a style of robotics that primarily uses simple analogue circuits, such as comparators, instead of a microprocessor in order to produce an unusually simple design. While not as flexible as microprocessor based robotics, BEAM robotics can be robust and efficient in performing the task for which it was designed.

https://en.wikipedia.org/wiki/BEAM_robotics

> ...computers can imitate all of human thought (not just one kind of it), since humans are physical beings.

Leaving aside the metaphysical questions it raises, we may well be able to build virtual humans someday by emulation of physics of the biology of the neurology of the psychology of people. It just might not be the most efficient way to do it, eh?

>chess was supposed to be a demonstration of human intelligence.

why would brute forcing tons of chess moves would be intelligence and not search. Aren't ANN just some kind of brute forcing, just that this time brute forcing happens at training time.

True intelligence could reason and adapt, the AI should be able to play any chess variant I invent without new training or if it is that smart should be able to defeat a human at any new task for both of them.

But yeah, many things that are not intelligent were called AI, like expert systems where the human had to define all the rules, or genetic algorithms where the hard part was made by humans by defining the encoding and the fitting functions, same for the ANN , the humans need to dot he hard part, collect good data , define the ANN parameters, then train and evaluate the results.

AI is so pathetic that the big software developers could not integrate it with coding, so you could do something like "Hey AI I need you to implement an integration with this API, find an SDK/library or implement it from the documentation and when my user uploads a photo use the xxx function of this API".

> Aren't ANN just some kind of brute forcing, just that this time brute forcing happens at training time.

Not by any definition of "brute force" that I know about. Brute force implies searching through all possibilities, which is impossible in chess as there are too many of them.

Not all possibilities, even basic algorithms will avoid testing invalid or obvious bad moves or do some kind of backtracking.

I did not studied this NN for chess, I am wondering if they just compressed the search space while training so searching is much faster.

How is that not what humans do?

People don't play billions of games to train. I think we have a mechanism to eliminate irrelevant inputs, focus on what is important and then we have the abstractisation and generalization tools. Then we can reuse same modules in our mind to play RTS games that are more complex then chess.

Anyway do you disagree that ANN are just a search/function approximation ? Or you think that when you played an RTS for first time the brain already had itself trained from genetics or previous experienced playing in the sand and you could win on the easy difficulty level without any training on millions of games.

> why would brute forcing tons of chess moves would be intelligence and not search. Aren't ANN just some kind of brute forcing, just that this time brute forcing happens at training time.

Right, so it's okay for humans to spend literally years training their biological neural networks, throughout their entire lives. And in addition to that spend thousands of hours honing skill such as chess (we'll call it "practice"). But when a machine starts from nothing and does the same, suddenly it's not intelligence, we don't call it "practice", we call it "brute forcing at training time".

> True intelligence could reason and adapt, the AI should be able to play any chess variant I invent without new training or if it is that smart should be able to defeat a human at any new task for both of them.

Sure, let's take chess, and modify the rules. At the start of your turn flip a coin, on heads: pawns can move left and right instead of forward. On tails: swap one of your own pawns with a pawn of your opponent. Will our chess grandmasters pick up this new game immediately without new training, or will they struggle initially?

Your requirements for artificial intelligence are much higher than for ordinary human intelligence.

I disagree. Get your best AI and make it play Minecraft, then same AI make it play a RTS, then make it play an RPG.

It will not work and a child will beat it at everything. The reason is we can understand stuff at a higher level, then we can create strategy like offensive, defensive, we learn from mistakes and don't randomly search for solutions.

About chess I bet a grand-master will beat your AI at chess if you swap those rules and you don't retrain the AI, you can only update the rules. The AI only sees numbers and functions, no goals, sub objectives etc.

It makes sense if your definition of "Intelligence" is "things only humans can do."

In which case the set of all things only humans do shrinks but never goes to zero and thus you can always maintain some kind of superiority.

IMO that's how people use the term - whether they are experts or not.

As long as we keep using the marketing term "AI" and we just argue about what the technical definition is, we're going to have this problem.

If Penrose wants, he can define AI using the Turing test. That's reasonable. I can say, no, that's moving the goalposts, my definition of AI is "things that looked incredibly hard in 1990. By that standard decent automatic translation is definitely AI." That's also reasonable.

As long as our definition of "AI" is fluid, discussing whether it's been met or not is pointless.

> I can say, no, ... my definition of AI is "things that looked incredibly hard in 1990

So, magic?

"Any sufficiently advanced technology is indistinguishable from magic."

— Clarke

>This sounds a lot like the usual moving of goalposts whereby "anything computers can do isn't AI, so AI doesn't work".

Well, it's also useful to establish a concrete definition of what is AI, and what is to be expected of it.

>"chess was supposed to be a demonstration of human intelligence"

Well, if it was, it was a bad one, and in restrospect it's obvious. You can play chess (as a program) and have totally 0 IQ in all over realms (e.g. any specialized chess engine, which is nothing like a general AI), whereas that's no true for humans. Human intelligence allows a chess player to ALSO play chess, it's not a chess-oriented algorithm.

The same way a plastic ruler might measure better than us ("this is 12.45 inches", whereas we might say "that's about a foot", but its "measuring intelligence" is not human intelligence by any definition.

> Well, if it was, it was a bad one, and in restrospect it's obvious.

That's exactly the point - this keeps happening with various things.

Having reasonable machine translation / playing chess / computer vision were all considered as AI problems at some point, now they're often dismissed as not being AI, depending on whether or not computers are seen as achieving them. Alternatively, progress in those problems is dismissed due to not having enough "understanding" or "abstraction", even if computers are way better than humans at solving them (e.g. Penrose's point about chess).

The goalposts keep moving so that AI is always what computers can't yet do.

I’ll bite. So what? Goalposts changing isn’t some sign of shadiness. Science isn’t a soccer game. Goalposts change all the damn time. Newtonian physics was considered as something that explained almost everything. Then we discovered that it doesn’t really, there are other theories that are better in being able to account for experiments. It’s not some sign of some big con. There was a goal, it was achieved and we moved on to the next one.

To use your physics analogy, the situation with Penrose and AI would be like someone going "see, Newtonian physics don't explain everything, I told you physics is unsolvable!".

>That's exactly the point - this keeps happening with various things.

Well, that's not "goalpost moving" then, it's "improving our understanding and updating wrong notions we had".

I think there is a deeper point here. I do AI research, I use logic programming, deep learning, and a bunch of other techniques. These things together can produce amazing results, solving problems of which humans could never solve in thousands of years.

However, I don't want black-box AI anywhere near me -- I don't want it deciding who gets into my University, the grades our students get, if I get a morgage or if I committed a crime. While AI can solve many problems, it's not really "thinking", it's just pattern matching and brute force, and (at least at the moment) every AI system is very easy to confuse if you try.

You think we're doing something better than pattern matching?

That seems to be what I do most of the time. :-)

I agree, the problem with AI is that companies seem much happier to say "the computer said so", and leave the answer at that.

Except of course when they want to get out of something, at which point they say "Oh, that was just the computer which said that, we didn't mean it".

> ` (...) His big concern about AI isn’t Judgment Day, but rather ‘that people will believe machines actually understand things’.

The part where I'm always stalled with these arguments: what makes you think people actually understand things?

I don't believe anyone with any significant knowledge of the subject has ever seriously suggested that the ability to play chess well requires full human level AI. Automatons that can play chess, some real and some fake, have existed for hundreds of years and nobody I'm aware of mistook them for human level intelligences but just clever curiosities, so I think this is clearly a particularly poorly informed straw man argument.

I can maybe imagine someone not working in AI or unfamiliar with how computers work, and ignorant of the history of Chess automatons maybe saying something like this, but even if so who cares? It's simply wrong. We're not going to issue Deep Blue citizenship because Chess.

The gold standard test for human level intelligence has long been and still remains the Turing Test. Not hobbled, limited "easy mode" tests as in some recent competitions for chat bots, but a full on, no holds barred freeform dialogue including whatever games, discussions, tests and topics a sophisticated tester chooses.

I'm not saying that chess equals human-level intelligence. I'm saying that dismissing any progress in AI because it doesn't solve a corner case of a previously important unsolved problem is moving the goalposts of AI research. Especially if dismissing the progress requires usage of fuzzy terms such as "understanding".

Right, but I suspect what’s happening is the inverse. Things like a chess and image classifiers are special cases. They’re important problems of course, but they’re not core building blocks of general AI and mistaking them for that is, well, a mistake. And sure some people do that, like the famous comment on Slashdot that Alphago was a sign that general AI was imminent, but actual AI researchers know this is simply not the case.

Discussions about be real human level AI do involve using poorly defined terms unfortunately, but that’s because we don’t actually understand general Intelligence. I think pinning down those concepts more concretely will be part of the process.

> The gold standard test for human level intelligence has long been and still remains the Turing Test

The Turing Test was an offhand example that Turing brainstormed. The hard part of the turing test isn't "intelligence" but "human" -- imitating a human's irrational quirks. It doesn't make sense to call that a "level", because computers are far better at imitating humans than humans are at imitating computers. In other words, humans can't even pass a symmetric Turing Test.

I disagree, I think the hard part of the Turing test is passing well constructed interrogations that check for comprehension, deduction, improvisation, etc. For example you teach the subject a novel game, ask them to play it, then change the rules, or you describe a situation, describe changes and activities and interrogate them on subsequent states. Essentially check for problem solving, deduction, etc, things that require actual intelligence.

You can't proxy a test for intelligence to other attributes, like quirks or personality. You have to actually test for the attribute in question.

> I don’t believe anyone with any significant knowledge of the subject has ever seriously suggested that the ability to play chess well requires full human level AI.

Douglas Hofstadter did.

> Automatons that can play chess, some real and some fake, have existed for hundreds of years

Those were midgets inside a box.

> The gold standard test for human level intelligence has long been and still remains the Turing Test.

Most HN readers, and most of Gen X, would fail a Turing test today as being "too ignorant to even talk to." So not sure what you're expecting.

> His big concern about AI [is] ‘that people will believe machines actually understand things’.

Right at the very beginning of the interview he has the same issue with human beings.

What it sounds like is that you have Penrose participate in game he wasn't party to ( not according to the interview anyway ) and then accuse him of violating the rules of that game.

The writing was not that great IMO. Would still like to read more about Penrose.

Personally, I enjoyed it very much. I thought the essay itself was put together like a set of Penrose tiles: it brought together the disparate aspects of Penrose's iagination and thought-life as if they were polygonal shapes -- all distinct, yet when arranged together they form a beautiful and very satisfying pattern.

Reply on news.ycombinator.com