Hacker News Reader

Comment by thomastjeffery | Hacker News Reader

The dumb criticize the blind.

What an absurd situation! How did we get here? Here are the steps:

1. Large Language Models have been presented as "AI", which personifies them instead of describing how they work.

2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content.

3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus?

4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!

View on HN

I'm not concerned about how they work or whether they're attaining personhood or not. I'm concerned at what people can and will use them for to the detriment of our civilisation

How they work is the most critical piece of that puzzle.

It isn't a mystery at all. LLMs implicitly model every pattern they can find in the text they are given. If the pattern aligns to language, humans categorize it as a feature. If it aligns to desired behavior, humans categorize it as a feature. If it aligns to undesired behavior, humans categorize it as a limitation. What humans categorize as features and limitations do not align with what the model is. A truth and a lie are structured with identical text patterns. The LLM can't categorize the two separately.

The only risk here is that people continue to personify LLMs, and place trust in their relatively limited ability to fulfill a human role. It's the same category of risk as bad security practices. The potential to make a mistake itself is not a danger: making the mistake is.

you certainly make an interesting point about fact and falsehood and how they interpret language; however, I disagree that this is relevant and completely disagree that personification has any relevance to the most immediate dangers

the danger here is that we make extremely powerful tools extremely available and people use those tools to hurt people, possibly a lot of people, possibly even by accident

the scary part is that hacker news is full of the exact people that have the knowledge and capability to do this, and look at the reaction to this post: a resounding fuck off

I just explained explicitly how and why an LLM is not, and never will be, an "extremely powerful tool".

A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.

have you tried gpt-4? how can you possibly say it's not an extremely powerful tool?

what happens when someone determined links it - or thousands of its - or thousands of next year's 3x as powerful its -- up to the internet with a code interpreter and a hard-coded poor intention?

I don't know; what happens? I'd really like to know what people are imagining it could do, especially if they've got some evidence for it!

it's a rhetorical question. think about it yourself. try it yourself

What could it do that isn't already being done? I ask this sincerely. We already have bots constantly attacking every server they find 24/7. We already have botnets

Is the concern that GPT will be able to find and exploit e.g. a novel RCE exploit in SSH?

I wish we could ask it to answer questions about GitHub repos.

For example, it could scan social media and use that information to send a very large number of people very personalized messages that manipulate them into doing X. Here, X may include meeting other people to tempt/coerce them into doing Y.

The reason this isn't possible now is that nobody has the time to look at millions of social media profiles and then write millions of personalized messages. To do it, you would need a lot of people who you would have to convince, including paying them. And even so, things like doomsday cults and terrorist organizations already exist.

Well, how can it be much worse than a smart human „hooked up to the internet“ with bad intentions? Other then generating comments or spam faster?

It's (to some subset of the audience), an extremely persuasive tool, which is powerful in its own right.

It's actual capabilities mean very little, if some people can be convinced into doing decision making without actually doing any decision making by it (or literally any tool), that's cause for concern.

Can you give an example of a concrete task that you believe that neither GPT-4, nor its successors, will be able to accomplish within the next 2 years? Keep in mind that ChatGPT Plugins[1] exist.

If you can't think of any tasks that a tool won't be able to accomplish within a 2 year period, I question your definition of "not extremely powerful".

[1] https://openai.com/blog/chatgpt-plugins

Categorize.

Decide.

Compare.

Refuse.

Do.

---

Sure, it can exhibit these behaviors, but each behavior itself never actually leaves the text that humans wrote it into.

That means we are reliant on how well the behaviors we want can be modeled into text patterns.

Unfortunately, natural language is ambiguous. There are many identical patterns that contain different behavior.

GPT's authors introduce some of these patterns as both "feature" and "limitation":

Features:

> The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.

Limitations:

> ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.

> given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.

These are not separate. They are two sides of the same coin.

GPT will never be able to look objectively at its own model. It will never be able to subjectively categorize patterns or tokens. If it could, it would not be an LLM: it would be an entirely new technology.

I am not sure what you mean by "each behavior itself never actually leaves the text that humans wrote it into". Do you mean that it cannot generalize outside of its training distribution, or are you gesturing at something else?

I notice that I am particularly confused by the example of "categorize" -- I am pretty sure you're not saying that ChatGPT can't categorize things, but then I have no good hypothesis for what you do mean.

> I notice that I am particularly confused by the example of "categorize" -- I am pretty sure you're not saying that ChatGPT can't categorize things, but then I have no good hypothesis for what you do mean.

That is exactly what I mean. GPT infers patterns. Some language patterns themselves are categorization, like words surrounding "or". It's important to recognize that humans are creating the categorization in these cases.

GPT does not differentiate at all between one pattern and another. It doesn't categorize the pattern that represents words, or the pattern that represents sentences, or the one that represents an answer in response to a question, etc. These are all symbolic categorizations that humans do to text; and we call that set of categories "language". GPT, ironically, has no concept of language or grammar. It only sees patterns as patterns.

Can you give a concrete example of a way that a human is capable of categorizing, where GPT is not? Some examples of concrete examples might be

- Humans can take a list of things like ["goldfish", "cat", "shark", "tree", "whale", "rock", "human"] and a list of categories like ["animal", "plant", "non-living"], and say which category each thing belongs to. GPT cannot do this.

- Humans can take a list of things like the above, say which categorization scheme is most appropriate for the thing they are trying to do, and then, according to the categorization scheme they said, say what category each of the things belongs in. GPT cannot do this.

- Humans can take a list of things like the above, and describe a novel categorization scheme for them. Humans can then say out which category things that were not in the original list belong to. GPT cannot do this.

- Some other thing where humans can describe how to categorize something but GPT cannot do so.

- Humans do something internally that GPT cannot do, but that thing has no observable effects from outside the system

Are any of these close to what you mean?

Yes, all of them. But we can go even simpler:

Humans can look at a list of words separated by commas, and call it a "list". We can also call it, "not a CSV file".

Humans can look at a 5, and call it five. We can hold up a hand, and say, "this many".

This behavior is named "semiotics". It's the act of using one thing to represent another.

LLMs are designed to intentionally avoid this approach. Instead of constructing an explicit grammar (like a parser), an LLM takes the opposite approach: inference.

Inference makes it possible to model the ambiguous patterns that natural language is made of. Inference also makes it impossible to define symbols.

Humans use both semiotics and inference. So far, it looks like no one has quite cracked how to do that artificially.

Portobello mushroom is certainly wrong, I also would have included color for salmon but that one is more up for debate I think. I think some humans would probably write the same categorizations though.

[[[Me]]]

> I will provide two lists of items inside brackets, separated by commas. I want you to categorize the entities in the first list into the categories in the second, keeping in mind that an entity in the first list may fit into multiple categories: [goldfish, shark, salmon, blue, tiger, cake, cyan, portobello mushroom, tree] [animal, fish, food, color, mammal, plant]

[[[GPT4]]]

> Here's the categorization of the entities in the first list into the categories in the second list:

    goldfish: [animal, fish, food]
    shark: [animal, fish, food]
    salmon: [animal, fish, food]
    blue: [color]
    tiger: [animal, mammal]
    cake: [food]
    cyan: [color]
    portobello mushroom: [food, plant]
    tree: [plant]

I'm not entirely sure I understand your argument but when you say:

> but each behavior itself never actually leaves the text that humans wrote it into...

I assume you mean that all the LLM can do is produce text so it's not inherently dangerous, but it's rather trivial to hook an LLM up to controls to the outside world by describing an API to it and then executing whatever "commands" it produces on its behalf and feeding the responses back to it. Wouldn't a setup like that let its behavior "leave the text," as you put it? I'm sure curious people are already doing experiments like this.

The LLM doesn't have to have agency to interact with the world in a way that mimics an entity that does have agency, that's what they are already doing with the text they generate, and why people are so quick to personify LLMs to say things like "they reason", "they understand", etc. If the text they produce, which mimics a sentient actor, is embedded in contexts that allow it to affect the world outside of the text interaction, there you have it, the LLM affects the outside world. Wouldn't you agree?

> I assume you mean that all the LLM can do is produce text so it's not inherently dangerous, but it's rather trivial to hook an LLM up to controls to the outside world by describing an API to it and then executing whatever "commands"

Yes, you can do that, but the result is guaranteed to be silly.

The LLM isn't conceptualizing what it reads. That was already done when the human writing it used language patterns to encode their own conceptualization as data.

Instead, the LLM takes an implicit approach to modeling that data. It finds patterns that are present in the data itself, and manipulates that text alrong those patterns.

Some of the LLM's inferred patterns align to the language structure that was intentionally used by the human writing to encode a concept into that data.

Humans look objectively at the concepts they have in mind. From that perspective, we use logic or emotion to create new concepts. If a human could attach their mind to API endpoints, there would be no need to use language in the first place. Instead of encoding concepts into intermediary data (language in text) to send to a machine, they could simply feel and do the API calls.

LLMs don't look objectively at their model. They don't have a place to store concepts. They don't feel or do any arbitrary thing.

Instead, an LLM is its model. Its only behavior is to add new text and inferred patterns to that model. By modeling a new prompt, any familiar text patterns that exist in that prompt's text will be used to organize it into the existing model. A "continuation" essentially prints that change.

When you attach that to API endpoints, the decision making process isn't real. There is no logically derived new concept to determine which API call to call. Instead, there is a collection of old concepts that were each derived logically in separate unrelated contexts, then encoded into language, and language into text. Those are just being recycled, as if their original meaning and purpose is guaranteed to apply, simply because they fit together like puzzle pieces. Even if you get the shape of them right (by following the patterns they are encoded with) there is no place in this process to introduce why, or to decide the result is nonsense and avoid it.

In short, the LLM can be made to affect the world around it, and the world can affect it back; but there is nothing in between it being affected, and it affecting the world. No logic. No intent. Only data.

> people use those tools to hurt people, possibly a lot of people, possibly even by accident

If it happens then hopefully it's by accident!

> I'm concerned at what people can and will use them for to the detriment of our civilisation

Maybe? But there's already all sorts of untapped resources for that - and we mostly just don't choose to use them. A lot of the misinformation and "fake news" that came out during prior election cycles in the US came from former Soviet bloc countries, and there is already significant untapped human capital in those places where it could be done for cheap, perhaps even cheaper than the cost of doing inference on premiere LLMs.

Meanwhile we see all sorts of efficiency improvements occurring for generally positive use cases from these LLMs, and it is hardly unique for technology to have some risks to it as well as positive benefits.

Things like GPT and Midjourney have been a qualitative leap over their predecessors from two decades ago, but they fundamentally are not doing something new - and even another qualitative leap wouldn't bring them to the level of existential threat that this letter seems to be afraid of.

I think there are significant and real concerns about AGI and similar, but I can't seem to find any reason to share these same concern around the types of "AI" we are seeing today.

> Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing.

Well, this is for honest reasons. The goal of a chatbox is to beat the Turing test. It has always been.

Those chatboxes didn't actually beat it, but it's clear that it's due to a technicality (they are easy to spot). They can do empty chats on the same level as a human.

(And so it turns up that the test wasn't as useful as it was believed to be at first.)

The problem is when you loop that logic around: it becomes circular reasoning.

What is the true source of an improved SAT score?

If it's a person we are talking about, then it's an understanding of the subjects being tested.

If it's an LLM, then it's...complicated.

It might be because the training corpus provided more matching text.

It might be because the training corpus provided text patterns that aligned better to the patterns in the SAT's text. The structure of phrases is just as important as the context they contain.

It might be because the training corpus had fewer text patterns that result in "a wrong answer".

Improving any of these means degrading the others. Logic is never involved. Symbolic reference, like defining words or "plugging numbers in" in to mathematical formula, is never involved. Doing well on one test does not mean doing well on a slightly rephrased version of that test.

This is the first time I have heard Yoshua Bengio described as "dumb".

According to the Oxford English Dictionary, the definition of AI is:

>the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.

Clearly GPT-3 fits that. You are welcome to have your own definition, but you can't just claim that everyone else is wrong just because they are instead using the standard one.

>2. Goals for LLM development were set for the personified attributes, and not the actual functionally of the real thing. OpenAI brags about how GPT4 scores at human tests: as if that has any bearing on the model itself, and not simply its content.

Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc.

>3. The success of an LLM at a personified goal is determined by magic. Does the black box output what I am hoping it will? How do I improve it? Weights? Better training corpus?

Not sure what you mean by magic, but it is certainly hard to predict what they can and can't achieve. I suspect if you asked a panel of experts in 2020 when we would see something that can achieve what GPT-4 was almost all of them would have been wrong, but I am open to evidence to the contrary.

>4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!

No serious person believes this.

> This is the first time I have heard Yoshua Bengio described as "dumb".

I was being cheeky: I meant it to mean "unable to speak". By feeding the personification narrative into itself, nothing is truly being said about what the model itself is and does: only the results of using it. Anyone who has based their understanding on that personified narrative is effectively "blind" to the true nature of the tech itself.

I admit I was also being rude, and that was intentional. To spend so much effort studying a thing, and base all of your conclusions on its output...I think it's absurd to then call oneself an expert. Expert of what? A black box you never even opened? The LLM itself can only perform inference, but we aren't held back by that limitation!

> the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.

That is an excellent definition for the field of science that is AI. "An AI technology" "Working in AI" These statements use "AI" as a category of pursuit.

It's not a great definition for an AI. Those words already define themselves. Artificial Intelligence: That means it was created intentionally, and it thinks for itself. No one needs to search a dictionary to know what "an AI" is.

If "Intelligence" is an adjective, the definition you provided fits. If "Intelligence" is the noun, then it better be "an intelligence".

> Yeah, evaluating language models is extremely difficult. Hence OpenAI Evals, etc.

I wasn't trying to say that it's wrong to evaluate what an LLM does when prompted with the text from a test. I think that can provide valuable data and insights.

The score we would give to a human taking the test is not valuable data. The behavior being measured does not align to any part of the LLM's process.

We don't have to approach this blindly. We can have an explicit understanding to base research on. That doesn't mean making assumptions about the results, but it does mean better understanding the process that leads to them.

>> 4. An LLM is understood to be a person in a black box, and we could not possibly understand what that person will do next. Danger!

> No serious person believes this.

Not literally, but narratively many do.

I feel more sympathy for that Google engineer that feel in love with their LLM. I am sure more people will become attached, at least once the rate limits are relaxed.

The rea trap is that LLMs can simulate empathy surprisingly well. If you have many problems to rant about but nobody willing to listen, the LLM is always there, it will never get bored or tell you to shut up, and it will always respond in encouraging and "positive" ways. Given how many people today do not have anyone like that in their lives, it's no wonder that they form an emotional bond.

I said thanks to each answer :)

Empathy is what language is for. It's language itself that is doing the empathy: and that isn't simulated at all. It was all written by people.

All an LLM can do is follow the structural patterns, and use them to shuffle.

This doesn't make any sense. Empathy itself is qualia; the language is merely a medium to communicate it, and far from the only one (e.g. facial expressions are generally better at it).

As for LLMs "following the structural patterns" of empathetic language - sure, that's exactly what simulating empathy is.

> the language is merely a medium to communicate it

Precisely. What else is communication if not empathy accomplished via language? Failed communication?

Language is not the only tool for communication, and communication is not the only tool for empathy.

> As for LLMs "following the structural patterns" of empathetic language - sure, that's exactly what simulating empathy is.

It's a matter of perspective: a character in a story follows the patterns of emphatic language. Do they empathize directly with the reader?

An LLM cannot break the fourth wall. It can only present one that was already broken.

An LLM doesn't look at the patterns in its model objectively. It only uses them in the process of modeling more patterns.

An LLM itself cannot reason, it can only model the patterns of reason that humans have already encoded into language into text.

I don't see what practical difference any of this makes. We can play word games all day long, but that won't convince Blake Lemoine or countless Replika users. To them, it's not "a character in a story", and that's the important point here.

The character of a story does not think, or do anything outside of what the writer of that story writes them to. The character cannot write itself!

That is the distinction I am making here.

Any person using an LLM is effectively playing "word games"; except instead of words, the game uses tokens; and instead of game rules, they follow pre-modeled token patterns. Those patterns are wholly dependent on the content of the training corpus text. The user only gets to interact by writing prompts: each prompt gets tokenized, the tokens get modeled, and the resulting difference gets printed back to the user.

No new behavior is ever created from within the model itself.

At no point in your rambling, incoherent comment were you even close to anything that could be considered a rational thought. Everyone on this site is now dumber for having read it. May God have mercy on your soul.

My point is that this fear is irrational: based on a misunderstanding about what LLMs can do, which itself is based on a misunderstanding of what LLMs are.

Is there a way I could have been more clear? Are you disinterested in the content of my comment, or the presentation? The only thing I can learn from your comment is that mine didn't live up to your mysterious standards.

https://news.ycombinator.com/newsguidelines.html

Yikes! Please don't do this on HN. I'm not going to ban you because we haven't had to warn you before, and also because I sense an overtone of good humor in the scorching remarks, but we can't have people posting like this—it leads to flames.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Reply on news.ycombinator.com