The Thoughts the Civilized Keep | Hacker News Reader

The Thoughts the Civilized Keep

noemamag.com

36 pointsarunbahl33 comments

>> Yet the connections GPT-3 makes are not illusory or concocted from thin air. It and many other machine learning models for natural language processing and generating do, in fact, track and reproduce real features of the symbolic order in which humans express thought. And yet, they do so without needing to have any thoughts to express.

Unfortunately there are plenty of people who seem to do that as well.

>> But the purpose of thought — what thought is good for — is a question widely neglected today, or else taken to have trivial, self-evident answers. Yet the answers are neither unimportant, nor obvious.

I thought the entire purpose of having a brain is to A) predict the future and B) plan for it ahead of time by C) making predictions about outcomes based on our various possible actions. The entire purpose of the brain is to predict and control. That's why projection is so important.

> Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated computation or behavior, no matter how clever.

Ah, right, it won't work because it doesn't have a soul. Got it.

I don't agree with the article, but hold on -- you don't need to buy "intelligence -> soul" for this.

There are plenty of things that can't (or can't efficiently) be represented in a single agent.

This is (for example) Hayek's main critique of central planning: "[price] understanding cannot occur in an isolated computation or behavior"; there are simply too many necessary details to fit into any one head/computer, and therefore we should prefer markets. (Markets being definitionally not located in any one isolated computation/behavior, but inter-agentially in lots of them.)

I don't see any fundamental reason a human brain couldn't be simulated at the atomic level by a turing machine, so it follows intelligence should be able to come from a machine, as far as it can come from a human.

Simulating that brain will require you to provide some sort of inputs and do something with the outputs. Which gets you into questions of "what does this simulated brain want" once again. How much of human desire and behavior comes from the body? I want to avoid pain (signals that my body sends to my brain). I want to experience pleasure (also signals that my body sends to my brain).

If you simulate a body with the same inputs as the existing brain, do you just have a "faster horse" problem, where you have a brain that could possibly perceive/process information faster (or slower) than a real human depending on the speed of the simulation? That could be useful, but sounds potentially rather bleak, especially if we limit what it can do to change its environment. It also seems limited for e.g. central planning - it can do planning faster than any individual, but likely will still have working memory size limitations, if it's just a simulation of a human brain. So it may not be so much better than parallelizing tasks across a bunch of humans.

More interesting, probably, then, than the "super human" style, is giving a new sort of intelligence different and additional senses, so that it doesn't want or need the same sort of physical embodiment we have, a la the "set sets" in the Terra Ignota series. But then we have to figure out what separates abstract intelligence from our own senses, and how to usefully augment it with new senses for a data-based world...

Yes, I often have to consider the context of the excel spreadsheets on my VMs, because you can’t know how to run a machine without knowing what it wants at a high level.

Given this is HN, I honestly can't tell if you are being genuine or snarky.

I don't see any evidence of dualism in the article. I think it makes sense that as people try to make intelligent machines we will find out more and more about what that means. The goalposts shift themselves until perhaps one day you find yourself there. But at every point you should expect it to turn out to be more complicated than you thought. Surely.

As for humans losing the ability to understand - that does sound a little bit panicky.

No, if you study how GPT-3 works, it becomes pretty clear that that model can't really produce "understanding", even if we can't clearly define what that is very well.

And I don't just mean this from the old point of view of "if we understand it, it's not AI anymore", I mean, the model has serious fundamental limitations. One of them is that it's static, so, once trained, it doesn't learn from its future input. Another is that when it is doing its "what is the highest probability text output to follow this text" (which, itself, is not exactly the sort of question "Strong AI" is the answer to), it by construction can only use a fixed-size window of text. So, for instance, it is by construction simply impossible for GPT-3 to write a novel, or even a short story, because the only way to get it to keep outputting is to provide it a window of the last X KB of output (I don't recall exactly how big it is) for the next continuation, which means that as it is writing the X+5th KB of output, it has literally completely forgotten the first 5KB of output, as completely as if it never existed.

On the flip side, it is clearly very interesting and there is something there. However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet).

> However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet).

I think that terms like "philosophical zombie" may be close to the truth, but I'm guessing some sci-fi work has a term that we could adopt.

> there is something there

There is indeed. I'll recount my own words after my first proper exposure to the model, in the form of playing custom stories in AI Dungeon:

"I played more of it today, and I can confirm that prolonged exposure makes me feel like I'm half awake, and half in a dream - particularly when the real world interrupts, I feel like I didn't fully wake up.

What's worse, whenever I tried to talk to a real person today, I've felt like I'm just feeding words to an AI, and anticipated "story" to evolve.

Never before has a game messed up with my mind so badly. I'm not sure if it's safe."

To which I got a reply perfectly summarizing the phenomenon:

"AI Dungeon feels a bit like a fever-dream."

It's been a year since; thinking back, I feel the trick here is, GPT-3 crosses the threshold of being able to produce plausible and interesting text - where by "interesting" I mean "almost, but not quite, unpredictable". The way other people and their stories are interesting - their words aren't totally random, you have this feeling for the possible trajectories the conversation can take, but you can't quite guess what they're going to say next. GPT-3 does that well enough that it's easy to suspend disbelief and feel as if you're dealing with a mind. But then the feelings that arise in you, that's just you projecting - engaging your mental machinery for dealing with other minds.

I did think of clarifying that the window is mostly large enough for AI dungeon to work. Though my understanding is that they do hit it if you go long enough, and the result is, well, exactly what you'd expect from that.

I did play with AI dungeon, and I would say the analogy to a dream is even deeper than it may appear at first, because I found that lucid dreaming techniques basically work on it, which is bizarrely impressive. I would characterize a lot of what one does in lucid dreaming (or, at least, what I do) is basically "expectations management"; if you can learn to expect something to happen in a dream, it frequently does. Not necessarily exactly what you "expected", but it will have a very significant influence. Correspondingly, it is possible (and indeed easy) to basically lead the AI along by using your text prompts to expect things to happen, and in a very dreamlike fashion, while that doesn't mean the exact thing you "expected" will be what the AI produces, it can clearly have very very significant impact on the result.

I should warn you this sort of removes the magic from AI Dungeon as an experience though... I have to consciously "let" it drive the story because it's pretty easy to exert much more control than you're "supposed" to have, when you know how to look behind the curtain.

> No, if you study how GPT-3 works, it becomes pretty clear that that model can't really produce "understanding", even if we can't clearly define what that is very well.

But the quote doesn't say that GPT-3 can't produce understanding because of the specifics of how it works, but "because understanding cannot occur in an isolated computation or behavior, no matter how clever", which is a more general claim.

And what is "understanding" anyway? Lots of "it can't be this it can't be that" but it doesn't tell what it actually is. Which makes it all easier to slap labels on things which don't actually need labels to just work. Does GPT-3 work? Yes it does and it works without bothering with whatever that "understanding" was. I bet Skynet wouldn't bother either to have an understanding or a soul, it would just... work.

I think it would be fair to say that "understanding" requires some sort of internal state change. We can definitely talk about how we did not understand in the past, but now we do. GPT-3 can't experience a state change, by its design.

I don't have to be able nail down exactly what kind of state change "understanding" is to point out that the way we use the word clearly entails one, and GPT-3 clearly can't have one, so therefore it is quite reasonably excluded from "understanding".

(In theory GPT-3 could be extended by continuous online training, but my impression is that convergence is too slow for that to be practical, and it wouldn't look much like "learning".)

It doesn't come with PURPOSE unless you force it to have some. Which is a perfectly fine avenue to take, I think. But without purpose, what you get is hallucinatory mockeries of meaning: trippy, but if it was consciousness you'd be looking for severe organic damage.

Purpose doesn't automatically mean civilized, but you can't get to civilized without purpose. It's an okay word to use for outlining what's missing.

AI output is increasingly coherent but MEANINGLESS. If you want to look to meaning and purpose, a better place to look might be emergent behaviors of the YouTube Algorithm, or Facebook feed.

Those can easily take toxic swings because they're operating in the absence of a higher sense of what will happen: they're the consciousness of a worm or insect. But they show purpose, where the more coherent and intelligence-like outputs of GPT-3 keep on betraying a lack of meaning.

GPT-3 doesn't WANT anything. There's the problem (though at this stage I think it's just as well). Only that will bring real AI… which can be a daunting prospect.

You're proposing that the... facebook feed is "more conscious" than GPT-3?

Seems like an interesting thought. The feeds have a purpose or agenda, actively influence real people in real life, and do so in a way that perpetuates their own existence. As long as they’re effective, they have real world institutions to support them. As it happens, they are perpetually learning and growing. Putting that next to GPT3, it’s like an intelligent rescue dog next to a horse that counts.

The facebook feed is at least an online system vs a static model, so YES.

Probably, yes. It's interesting how consciousness could be a matter of drawing boundaries.

A well-known example are the ants: individual ants are somewhat dumb, barely above simple chemical seekers. But the ant colony behaves as a single organism, and quite smart at that.

Other examples include cities, nations, corporations - it kind of feels like these organizations have minds of their own, with individual humans being cells, and the bureaucracy forming a computational substrate.

In the same vein, Facebook feed could be seen as a mind of its own. So could the entire human civilization.

Yes. It's bound to be a very complicated web of intentions and subroutines to produce outcomes, very likely with no one human overseer. It's gonna be a big pile of messy dumb code, not necessarily self-modifying but very likely with attempts to 'learn' and adapt on the fly towards basic ends.

It will be HUNGRY in the way GPT-3 is not: through both indirect and direct pressures, it will be struggling to maximize engagement, reach more places, get more food in the sense of people trying to interact with it.

The Facebook news feed, and even more so the less human-centered algorithms of Google and YouTube, are more conscious than GPT-3. When they're adjusted, such as the times Google has fiddled with the algorithm because YouTube was pipelining people to the alt-right after it 'observed' that this heightened engagement a lot, the algorithms are adjusted by trying to give them other types of information, other priorities (like giving them more sensory organs to highlight stuff they'd missed about their larger environment) rather than by diving in and recoding specifica about particular youtube creators, or looking for a more narrowly defined 'politics' that had been coded in there.

No such nefarious coding existed. The algorithm is alive, but blind. It functions on a scale where it can't NOT trample its surroundings, so it must be given eyes… and the consciousness gradually increases.

It’s wrong to say it doesn’t want anything, it’s solely motivated to minimize training error.

If you align what minimizing train error means with a larger purpose, it will seemingly make magic happen.

Just like you, online, trying to figure out how to turn this into copulatory success. It doesn’t matter if you succeed but don’t understand why, only that you succeed.

Is it motivated or is it just what it does? Is your thermostat motivated to minimize the temperature difference in a particular direction as well or is it just a machine that does that? Is your bicycle motivated to stay upright when you ride it or is just a consequence of physics (and halfway decent balance)?

Are you motivated to "find love?" Or just mechanistically following your genetic drive for reproduction?

It's a neocortex that doesn't have the rest of the brain. : p

GPT-3 is a moderately proficient author with nothing to say. The problem isn't the proficiency, it's the lack of content.

Some of us here despise the writing style of The New Yorker. The writing is beautiful, literary, but it takes it forever to say anything. GPT-3 is like that, but worse, because it has nothing to say.

No, the argument is that it won't work because it doesn't have a _body_.

> Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated computation or behavior, no matter how clever. Understanding is not an act but a labor. Labor is entirely irrelevant to a computational model that has no history or trajectory in the world.

For more along these lines: https://en.wikipedia.org/wiki/Situated_cognition

> Civilization advances by extending the number of important operations which we can perform without thinking about them.

I can't help but read this as "automation".

I think there is an interesting philosophical overlap between automation and intelligence. Perhaps some virtuous cycle exists between these things. Like, consciousness is probably only feasible because so much of what happens in our biology occurs without actual intent. If you had to actively process the dilation of your blood vessels or digestion of food, there wouldn't be much room left over for other higher functions.

You see this in tech too. For instance, build automation makes it feasible to then go and do certain things that otherwise would have seemed too complex or costly if someone was baking software by hand.

> I think there is an interesting philosophical overlap between automation and intelligence.

On the flip side - there's an interesting hypothesis that the purpose of consciousness is to mediate conflicting non-conscious impulses. An example would be falling in the water - you have to mediate the impulse to breath with other conflicting impulses (such as the knowledge that you will drown).

So I think consciousness/intelligence helps build skills and push them into the subconscious (thus 'automating' them) and then consciousness has to step in again for edge cases the automation can't (yet) handle.

There's some fun discussion on the mediation hypothesis (with some Dune tie-ins) here: https://www.rifters.com/crawl/?p=791

Why does this read to me like an overview of a D&D module where our intrepid PCs fight their way to the emotionless lich king at the heart of the keep? Is it the language use? The priming effect of the image?

> For AI researchers to move past the behaviorist conflation of thought and action, the field needs to drink again from the philosophical waters that fed much AI research in the late 20th century, when the field was theoretically rich, albeit technically floundering.

Well, gosh. Fancy a professor of philosophy deciding that AI needs more philosophy. Clearly, if the price of making progress in the field is to engage in all this ugly mathematics and -eww- engineering, then it's no price worth paying.

>Clearly, if the price of making progress in the field is to engage in all this ugly mathematics and -eww- engineering, then it's no price worth paying.

What makes you think the author is saying anything like this?

Reply on news.ycombinator.com