Hacker News Reader

Comment by bryanrasmussen | Hacker News Reader

>“That language models can do these sort of things was never discussed in any literature that I’m aware of,"

I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system? Based on layman's readings in chaos and complexity theory.

View on HN

Same here. I don't think it's surprising, but depending on where you say it, you'll find people insisting that this can't be possible.

I think a lot of our widely held cultural beliefs on this front have been informed by academic philosophy from the 60s, 70s, and 80s. In particular, I would go do far as to say that Hubert Dreyfus, author of the book "What computers can't do" and frequent friendly adversary of Daniel Dennett, has a lot of responsibility here. He is famous for claiming computers would never become competitive at chess because chess required "insight", and always differentiated computer capabilities from complex behavior or human-like behavior, because the latter had something special that could only be represented with vague and underdefined terms. Even now I think Dreyfus is approvingly cited, while his core thesis is getting closer to being refuted every day if it hasn't been already.

One thing that LLMs have made me realize is just how ungrounded a lot of mainstream academic philosophy was in the 70s and 80s. For example, so much of Derrida's work centered around the impossibility of truly communicating shared meaning between individuals through language. The fact that we can now communicate so effectively (with remarkably few contextual errors and razor-sharp conveyance of intent) with an entity/technology that is not even human pretty much destroys so much of Derrida's oeuvre as false and delusionally solipsistic, basically just a silly game. He had his critics who argued the same thing, especially analytical philosophers, but they never had as much purchase as he did in the academy.

I haven't read Derrida in decades, but your post inspired me to ask chatGPT about this. Version 3.5 would have none of it, and was adamant that Derrida's views were in no way threatened.I almost got the feeling it wanted to call me a bad user just for asking! GPT4 on the other hand, went into a long explanation about how its existence challenged some parts of it by providing analysis of concepts like différance, trace, and undecidability. GPT4 is great at discussing itself and how LLMs in general fit into various philosophical debates.

Version 4 is definitely better in discussing philosophy in general. 3.5 was able to summarize philosophical material, but once you started to engage with ideas critically, it would tend to get into largely nonsensical moralism about needing to make a good faith effort to understand individual philosophers' ideas. There's much less of that in 4.

>The fact that we can now communicate so effectively (with remarkably few contextual errors and razor-sharp conveyance of intent) with an entity/technology that is not even human pretty much destroys so much of Derrida's oeuvre as false and delusionally solipsistic

Isn't the LLM just good at giving us the illusion of such effective communication? How can we have true "shared meaning" with a device designed to simulate the experience of shared meaning? Isn't the fact that it's 'mind' is truly unknowable and unrelatable a demonstration of Derrida's point?

If you can tell it things in a way that it successfully acts on that information, then that seems like effective communication.

When I use GitHub Copilot, I'll often write a comment describing some code I want it to implement and then it spits out a suggestion. Occasionally it fails badly at this and I realize I worded my comment badly. When I reword the comment better and then find that Copilot's next suggestion nails it what I wanted, is that not the result of me effectively communicating with it through the comment? (If I did that with a human coworker instead, is that communication?)

This is a long running strain of criticism against continental philosophy. I don't mean to write this to refute you, just that many people do think that the continental philosophers are tilting at windmills and aren't really based around objective reality in the way a lot of earlier (pre-Hegel) Western philosophers were. There's also people who would claim that Derrida's linguistic relativism is in fact bolstered by LLMs.

> I had previously the expectation that unpredictable emergent behavior would exist in any sufficiently complex system?

Yes and no.

It's reasonable to expect that some unpredictable emergent behavior would exist in any sufficiently complex system, but it's not warranted to expect whether a particular capability will or will not emerge out of that system (that's the unexpected emergent behavior part); and it also was debated whether language models are a "sufficiently complex system", as many thinkers asserted that they should encounter fundamental limits in capabilities.

For some reason that quote and your point made more salient to me a trend with AI research, which is that it seems to increasingly become difficult to understand and predict. That is, for awhile it seemed like tinkering in computer science without strong grounding in mathematical or statistical theory, and then there started to be a lot of discussion about "black box" processes, explainability, and interpretability and not fully understanding what's been done, and now the discussion is about not being able to predict the features of the models.

I'm having trouble putting into words what I'm thinking but this whole field increasingly seems to have moved from something very theory derived into something very empirical very quickly. I wonder at what point gains will start to decrease only because people won't know where to go with things anymore.

In my mind it’s not dissimilar to when new physical phenomena were discovered in the past (electricity, electromagnetism, steam engines, atomic spectra, invariance of the speed of light, blackbody curve), where it could take decades to come to a proper understanding of how and why they work.

You're right, but the distinction between emergent and useful is important.

Writings on chaos and complexity theory obviously aren't talking about LLMs. Those theories are so high level that it might as well be akin to "philosophy" to the applied scientists working on LLM research.

Additionally keep in mind emergent behavior is a very rare occurrence in even the most complex software projects. I mean it's common if you count "bugs" as emergent behavior. But emergent behavior that is a feature who's origins are not completely understood? That is extremely rare.

OP has a good point I think, even if it does not refer to LLM, which to me is too strict of a requirement.

I think emergent behaviour happens in a lot of videogames. Famously in Dwarf Fortress, with the cat getting drunk, but also in general, where game designers play the game to see if emergent behaviour of the game rules "feels" good.

Yesterday I was reading a book about designing games, and it literally has a section called emergent behaviour.

If by emergent behaviour we refer to something like Ghost in the Shell, then it happens less often :)

>I think emergent behaviour happens in a lot of videogames. Famously in Dwarf Fortress, with the cat getting drunk, but also in general, where game designers play the game to see if emergent behaviour of the game rules "feels" good.

Depends. Dwarf fortress and games are sort of a contradiction. Emergent behavior is emergent because the behavior was not designed explicitly. However for games like DF the game was explicitly designed to have "emergent" behavior when the definition of "emergent behavior" is for the behavior to have NOT been designed.

Don't get too hung up on that concept though. It's just a contradiction in English vocabulary there's no deeper underlying meaning behind that other than a semantic language issue.

Anyway my point was emergent behavior in software is rare because we're operating in controlled environment. It's rare even in games. It's not an expected attribute at all. I'm not saying this isn't interesting to think about, but the comment I responded to was in fact, factually not fully correct. Emergent behavior is NOT expected. But it does happen, in the case of DF it was "designed" to happen, and it has happened elsewhere as well.

Usually though when it does happen it was explicitly "designed" You can see this in genetic programming or evolutionary programming especially.

My thought of what "emergent behavior" is, isn't that it is necessarily "unintended", but rather that it isn't really present in the small scale stuff, and isn't an obvious consequence of the small scale stuff? Like, a qualitative difference between the base level stuff, and the consequences when it all fits together.

Like, if God intended for the universe to have something which acts like water, when designing subatomic physics, that doesn't make the behavior of water "not emergent behavior".

That's a blurry definition I can sort of feel what you're trying to convey here but objectively it doesn't make sense.

A brick building for example. Is a brick building the emergent effect of bricks? No. This makes your definition inconsistent because it should fit the brick building but colloquially we know it doesn't fit. A brick building is NOT an emergent effect of bricks. It was deliberately designed from bricks.

So long as the effect is some reduction of entropy and there was no intention in producing such effect then it is "emergent." This is really the only definition that is consistently inline with our intuition of the concept.

At least in the social sciences not so abstract. There are certain behaviors that can just be explained as emergent, rather than individual behaviors. That might be crowd, market, group, politics, culture (with art, language, fashion, taboos, etc.).

Of course. But the computer is a controlled environment designed to imitate the rigor of formal mathematics. It is usually not expected for such behavior to occur in computing.

Of course in the real world it happens all the time. YOU are emergent behavior. Humans and the rest of biology come from natural selection and evolution, which is basically by definition, emergent behavior of atoms.

Reply on news.ycombinator.com