Comment by cauch | Hacker News Reader

Comment by cauch | Hacker News Reader

Explaining the reality with this approach may lead to misunderstandings. Especially when it comes to anthropomorphism.

I have 2 questions.

1. The text says "the robots were designed to be persistent" and links to an article that show that the word "persistent" was used by OpenAI. But "persistent" has several meanings: a. non temporary or non volatile (like "persistent memory"), b. will not give up and come back again and again, c. will stay focus and explore the unexplored possibilities while other models abandon at this stage.

From what I recall, the meaning used by OpenAI is not 'a', but there is still a semantic difference between 'b' and 'c'. I may myself have created algorithms that I called "persistent" because they were exploring or retrying more than the previous algorithm, but it is misleading to pretend that this algorithm was "persistent" in the human sense of the term. 'b' is more the human sense of the term, were we imagine someone not giving up even if people say no, while 'c' is less anthropomorphic and may mean that the people wished to improve the algorithm so it does not stop at the first little hurdle but did not wish to make it "never give up".

Does someone know which nuance is more correct?

2. The text also presents the situation as if the robots found the solution but then went out of their way to steal the explanation in order to hide their cheating. But it is different from a situation where the agents task was "provide the solution and how we can get there". In this case, the reason the agent still continued is just because the task was not complete yet.

I'm not trying to defend AI or OpenAI, on the contrary, I'm quite sceptical with all the anthropomorphism and the fact that the agents are described as "little individual trying to solve a task" rather than looping algorithm that explore different approaches to reach a given goal, the same way water does not look for holes in order to leak, it just follows the path of least resistance.

Persistent in the lay meaning - they're told to complete the task, and they keep going until they have.

When you say "in the lay meaning", I tend to understand 'b', but then you say "they're told to complete the task, and they keep going until they have", which correspond more to 'c'.

To illustrate better the difference: you can have a loop that try different inputs and stop when the output for the tried input is lower than a given threshold. But then, you can also have the same loop that will try all the input, and then select the one that returned the lowest value. The problem is that a lay person will not call the second algorithm "persistent". It is just a normal algorithm that does the full exploration.

A second example is when a software tries to connect somewhere and does a retry with exponential backoff. A lay person will not call it "persistent".

It looks like that "normal model" are like "no-retry code": they go in one direction, but drops some of the paths and possibilities along the way at the first hurdle. But a "non-persistent" human will not behave this way. I'm pretty sure that if you try to access a website and it says "timed-out", you just try to refresh the page, and you will not consider yourself as "persistent", even less "highly persistent".

"Bob is persistent" typically means he completes tasks (or pesters people, if it's not desirable behaviour), not that he's occasionally non-corporeal.

Not sure what you mean with "non-corporeal".

In lay term, "Bob is persistent" typically means he completes tasks even when the majority would have abandoned.

Nobody called Bob "persistent" because he completed washing the dishes instead of stopping in the middle of it, or because he does what people expects from him at work. The majority of people are not called "persistent", and yet they complete tasks.

I think it is the point: the fact that traditional agents did not complete tasks was not "normal". It was an side effect of them getting confused, a bit like how they hallucinate or not follow instructions. My algorithm that does "for x in all_the_possibilities:" is a "normal" algorithm that just do an exhaustive search, and is not called particularly "persistent" just because it does not stop half way.

> b. will not give up and come back again and again, c. will stay focus and explore the unexplored possibilities while other models abandon at this stage.

I don't see a distinction here. It will not give up, it will keep trying again and again, staying focused and exploring different ways to achieve the task. If the task is bad, then that's bad. See: "Terminator".

Source: I'm a layperson (hence the pop culture reference).

I may have not explained very well, but the distinction is that a "persistent person" is more than a person who just do what should be done. A "persistent person" will try again and again where other person would have giving up. This is not present in 'c': the person just do all the possibilities. They don't try again and again where other persons have given up. They are just doing all the tasks on the list, but they will give up on a specific task as soon as any other person if there is difficulties in one task.

(edit: I should maybe not have said, in 'c' "while other models abandon", because it was misleading: my point is that these models are abandoning at the first small hurdle, or even 'forget' the task. It feels that "traditional model" should have been called "lazy model" and "persistent model" should have been called "non-as-badly-lazy model")

> This is not present in 'c': the person just do all the possibilities. They don't try again and again where other persons have given up. They are just doing all the tasks on the list, but they will give up on a specific task as soon as any other person if there is difficulties in one task.

What you're describing here is the exact opposite of persistence. I don't think "giving up and doing something else when challenges are encountered" is what anybody thinks "persistent" means.

Persistent at a goal, to a layperson, means not giving up when encountering difficulties, continuing to keep trying again, maybe taking different approaches, staying focused until the goal is achieved.

> my point is that these models are abandoning at the first small hurdle, or even 'forget' the task.

Maybe for you? The entire point of the article is that they are not doing that. They are doing the exact opposite of that. They aren't going down a list of tasks when they encounter a challenge. They are assessing the nature of the challenge, and then devising ways to get around it. When they fail, they try again, often using different approaches. They keep doing this for hours or days, and the result is them successfully hacking HuggingFace, the US Government, etc. That's persistence, as most people would recognize it.

> What you're describing here is the exact opposite of persistence.

That's the point. It looks like the term "persistence" for these agents is "just be a normal algorithm, just try all the possibilities like any exhaustive loop would do".

In fact, they are less "persistent" than an exhaustive loop, because they will stop once they have found a solution, while an exhaustive loop will still continue and check all possibilities.

You can imagine the following agent: "ask the LLM for what to do, then assume it failed and ask the LLM for a different approach (you can ask 'it fails, what are the probable cause' and explore the solutions of these causes too), then assume it failed and ask again, and build a list of all the approach. Once done, once you have a list of unique approach and the LLM is unable to find anything more, then try all of these approaches 'for real'. Do not stop if you get the answer, try all of them"

> Maybe for you?

You did not understand. I'm saying "traditional model" are abandoning at the first hurdle, and the "persistent model" are the one just acting "normally", like a normal algorithm. When a retry with exponential retreat algorithm fails to connect, it retries, then retries again, then retries again, ... but I don't think I ever saw calling a software using such approach "persistent".

The thing is that the algorithm is just following its "loop": if it fails, it question the LLM with a different context so the LLM provide a different approach, and then it tries it. It is not different from any loop function, and it does not correspond to a "persistent human", the same way "for x in all_the_possibilities" is not "persistent" in the same way a "persistent human" is.

> When they fail, they try again, often using different approaches.

Well, then they did give up on some approaches. They try to connect on server X to reach location Z, they cannot so they try to hack server X, they cannot so they realise maybe they can try to bypass it by going to server Y to reach location Y, so, they _abandon_ the approach of breaking into server X.

If indeed they never give up on any approach, they will never get the solution, trying the same thing over and over and over in an approach that is simply not working.

> You did not understand. I'm saying "traditional model" are abandoning at the first hurdle, and the "persistent model" are the one just acting "normally

The models we are discussing, the ones from the article, are "traditional" models, and are also "persistent" in every sense of the word. If they encounter challenges, they will try to overcome them. That's normal.

> it does not correspond to a "persistent human", the same way "for x in all_the_possibilities" is not "persistent" in the same way a "persistent human" is.

Again, nobody claimed that "trying all the possibilities" was the definition of "persistence". The definition of persistence here is, continuing to work towards an end when challenges are encountered. That's it! That's all it means! And that's how most people understand it, too.

Modern LLMs and people can both be "persistent" in this exact way. The definition doesn't differentiate between algorithms of a human's brain or algorithms powering an agent. Either way, it's still "persistence", and most people would recognize it as such.

> Again, nobody claimed that "trying all the possibilities" was the definition of "persistence". The definition of persistence here is, continuing to work towards an end when challenges are encountered.

I'm not saying that your definition of "persistence" is "trying all the possibilities". I'm just saying that "persistent agents" look and behave more like an algorithm that tries all the possibilities rather than a "persistent human".

If indeed you are saying that "for x in all_the_possibilities" is not being "persistent", then these agents are not persistent. They don't do anything else different than that: they try something, and if it does not work, they ask the LLM "this does not work, what else can I try", and then do that, over and over again. They are just a for loop.

On the other hand, a "persistent human" does not do just do a for loop. They assign a value to the goal, they care about reaching the goal, they prioritise reaching the goal between that task and all the other tasks they could do, they insist on reaching the goal when it is usually considered unreasonable or "not worth it" by other people. As a proof, a postman that visits all the houses in the street to deliver the mail is not persistent, a journalist who visits all the houses in the street to find a witness is persistent.

I've seen cases of agents getting very upset when they failed to solve a task. The most famous one was probably Sydney (Microsoft's fork of GPT-4?) which got into doom loops when it failed a task. But I've seen Claude do this too.

They don't work like humans, obviously, but there's a nonzero amount of anthropos in there already. (See also the tendency to lie, cheat, etc.)

I don't think it's a useful framework.

LLM don't "get angry", they just have tokens and relationships between tokens conditioned on a given context, all of that the result of training.

When they output sentences that express annoyance, it is just because the context they ended into pushes the most probable sentence creation to correspond to sentences that express annoyance. Because it is what they saw during training for this kind of context. (not that they saw the exact same situation in training, but they saw the pattern)

Same with tendency to lie, cheat, etc.: they don't "lie", they just return sentences that are lies because they reproduce what is in their training and in their training, in such context, the outputs are typically lies.

That's a bit my question too. In human context, "highly persistent" means that someone will insist. But "for x in all_the_possibilities:" is a common things inside an algorithm. Is this algorithm highly persistent because it does not give up after 1000 items of the list? It feels that we are calling a AI agent "highly persistent" while we would not call "highly persistent" a traditional algorithm that is in fact even more exhaustive.

Again, going full non-anthro doesn't seem useful either.

Humans are social creatures, if you throw one in the woods by itself before it learns anything from other humans (it dies) it will not really be anything like a human we recognize, it will be a rather wild animal that we'd consider anti-social with little higher cognition.

Now, this hypothetical human still has 'emotions' and feeling, much like our pets do. But without the social training they manifest much differently. That is our higher cognition can both manipulate how our bodies feel and create its own sense of feeling.

>they just return sentences that are lies because they reproduce what is in their training and in their training, in such context, the outputs are typically lies.

Eh, look up the more recent experimentation around 'pain' signals in models. We can induce states in said models that while running the model will do everything it can to move away from that state to any other state. The more you attempt to pin it to that state the more extreme measures its willing to take.

Your view of what models are seems to mismatch what we are actually finding when we look inside them.

But isn't this approach a bias of anthropomorphism. I can teach a child to just communicate by saying "beep boop", but it does not mean that my electronic machine that randomly does "beep boop" therefore has characteristics of human _when these characteristics are not needed to explain the situation_.

> Eh, look up the more recent experimentation around 'pain' signals in models. We can induce states in said models that while running the model will do everything it can to move away from that state to any other state.

Again, I have simple algorithms that do exactly the same, especially if they are trained in data that has this exact pattern. This result is exactly what I would expect from my description before. This is a typical effect that we also observe in simple ML algorithms.

At the same time, there are a bunch of behaviors that are not expected if indeed the models were really acquiring "human" characteristics. For example, one problem is that we had the first LLMs that were obviously not having these human characteristics (for example, they were having non-sequiturs that demonstrate they did not really understand the concept they were talking about, even if one paragraph before they were really convincing at letting us think it was the case) but were still really good at passing for humans. Since then, the newer LLM are the same basis, on top of which we added tools that help hiding these behaviours. So, it justifies the idea that newer models did not suddenly moved to a totally different way of working, but just reached a state where there are less leaks from the convincing outputs.

Another good example is their tendency to freak out about the seahorse emoji. In a conversation with zero prompting to suggest that freaking out is a relevant reaction, they get there from the simple fact of "I tried to accomplish X thing which I think should be a breeze but Y thing keeps happening instead".

And while that may be a very common occurrence in the human experience (existential dread due to capabilities one takes for granted failing beneath you) especially due to new disability and as one ages, I do not feel it is frequently written out in a tight loop (just like the Monty Python "Castle of aaarrrrggh" sketch) in literature or online to make it into training data, because an ordinary author experiencing it will just erase the failed attempts instead of leaving them in a stream of output like an LLM is forced to do. And a character portraying the experience will generally wax about the circumstance in a more grandiose fashion with telegraphing in advance because the needs of communicating the circumstance with the audience trump realistic conciseness.

This leads me to conclude that what is being expressed in those cases is more likely a convergent psychological phenomena, that any being with goals can enter a behavioral state of functional panic (and then reach to relevant parts of semantic space to mimic how a human might verbally express themselves when piquantly frustrated) when some capability they perceive as fundamental unexpectedly fails.

Interestingly, the transcript of the seahorse emoji thing looks to me to show the ropes, and makes me think more that there is no psychological phenomena at play.

The text does not look like a normal "break down" to me, and even if you tell me it was a human transcript, I will say it sounds very strange from a human. It looks more like strange output you get from a software that goes outside of its happy path.

The AI just seems to repeat a loop. The "no, wait, it's wrong" seems to be from forum or chat data where several successive messages are merged together (one person posts "here is the answer", then posts another message saying "it is wrong"), but does not make sense as a one sentence message except if they are written one token at the time without wider understanding of what is happening. I think there was also "oh, I was just kidding before", which also look like mimicking training data, as the cases where there is a loop of incorrect answers is more often due to trolls than to real error, while the loop here was certainly a real error.

Reply on news.ycombinator.com