Hacker News Reader

Comment by spywaregorilla | Hacker News Reader

Seems like a nothing story. Just looking at the game, there's obviously a constant decision to be made of chase more sheep or instantly die. It sounds like in the original model they had a max of 20 seconds, so it's not surprising that you would just tank your losses to maximize your score every now and then.

Anyone who tries to devise optimal strategies for things should be able to see this isn't especially interesting.

Social metaphors are wildly out of place.

They say "unintended consequences of a blackbox" but I doubt that's true. Make it a deterministic turn based game and run it through a perfectly transparent optimization model and I wouldn't be surprised to learn this was just the best strategy for the rules they devised. I really hate when people describe an ai as something that cannot be understood because they personally don't understand it.

View on HN

Exactly, from technical perspective it's a nothing story.

It's interesting, though, how strong of a reaction general public had to this. The story must have strongly resonated with what some folks were already feeling. When you squint (pretend to understand the technology not at all) it's a tragic story. The situation of the wolf seems similar to the situation of some people. Chasing their careers in a highly structured, sort of dehumanized, environment of constant pursuit. "Supreme Intelligence" (that's what a layperson may think of AI) looks at a situation of the wolf and decides that it makes no sense to continue the pursuit. Moreover, what is "optimal" is the most tragic result - suicide.

> The story must have strongly resonated with what some folks were already feeling.

Yes, because we don't see things as they are, we see them as we are.

At a Grateful Dead show in Oakland this geezer said to me:

Your perception IS your reality man!

Schopenhauer: World as representation.

I'd have loved to have been arond to a Dead show! I know it sounds a little ungrateful coming from someone who lives in a period of unprecedented access to all kinds of wonderful music being written all the time, but there's something about the Dead that really connects with me that I can't quite put my finger on.

Dark Star Orchestra is your current best bet https://www.youtube.com/watch?v=y8_THRZLSi4

Shrug. Another way to frame this is a poker bot learned to fold when given a bad hand, and they only gave it the same bad hand.

Yes, yes, woe is the individual in modern capitalist society but the only reason people are reacting to this are that they don't understand it and they've been told it's something much more emotionally impactful than it actually is.

>but the only reason people are reacting to this are that they don't understand it

I think it's much more likely that they're reacting like this because they see their own plight in the wolf. It doesn't matter why the wolf killed itself, it became a meme that allowed many Chinese to reflect together on a common plight.

I think there's a bit more to the analogy than just the suicidal wolf, though. The wolf is offing itself to minimize loss because there's no clear path to a better outcome.

This seems like a common refrain when we see radicalized engineering students from less-developed countries, who are notably common in extremist groups. They're people on a very difficult path (an engineering program!) with no real path to success (living in a society where unemployment for people with degrees is very high). Cost for continuing on the path is high, and there's no obvious path to get the good outcomes.

Having reread the article, it seems like the concept of suicide doesn't weigh into the cultural reaction at all. It's just giving up on the chase.

Yes you are quite right. The social media reactions did not suggest an attitude of suicide at all. It was more of laid back life instead of meeting expectations and attaining so-called success.

From the article in contrast to what you said;

> Perhaps the true lesson to be learnt here isn’t about helplessness and giving up. It’s about getting up, trying again and again, and staying with the story till the end.

I find the possibility of contrasting interpretations absurd. The problem with using any dead matter for our meaning making needs is it is ultimately a self-referential justification for how we think we should feel, while being equally or even more prone to self deception traps.

AI being the object is irrelevant here, this is nothing different than astrology or divination from tea leaves etc. It is 2000 BC level religious thinking with new toys.

Any programmer would have seen the issue and made the change about rewarding suicide.

The ONLY reason this was written was because the researches hired a programmer to make a specific thing, then is was too expensive for them to make more changes so they published the mistake.

Similarly I've seen A LOT of people posting stories about "chat bot exposed to internet started praising Hitler and became racist/sexist/antisemitic" as a proof that "supreme intellect sees through leftist political correctness and knows that alt-right is correct about everything".

I think a lot of those people are joking?

It's really not that deep, people will always find sport in scandalising people with a stronger disgust reaction than themselves. It's more a new way of teaching a parrot to say "fuck" rather than a heartfelt statement of political belief in my opinion.

> It's more a new way of teaching a parrot to say "fuck"

This is a excellent analogy for this sort of behaviour, thank you.

Exactly. It is a social commentary story where a result from a student's project was a lucid analogy of the plight of their lived rat-race in modern China, with the lesson being: Cut your losses and lie flat. To those within ML field, this is less than new, but as a commentary on how such ML issues can be a teachable and easily understood analogy to people's lives certainly makes the story interesting to me.

Spot on. Funny results from poorly specified AI experiments have cropped up since the 90's. But the interestingly angel her is how this one came out of nowhere at the right time and resonated with young working class Chinese.

> Exactly, from technical perspective it's a nothing story.

I think that one thing it points to is how technology can discover novel iterations on a system. Imagine if this was a system modeled around a network and the agent was trying to figure out how to get from the outside to read a specific system asset. With the right (read: very detailed) modeling you could create a pentesting agent.

> Chasing their careers in a highly structured, sort of dehumanized, environment of constant pursuit.

They have a word for it over there: involution i.e. no matter how much effort you put in, you get the same result.

It's not surprising from the perspective of an "AI actor". But if you call it a "wolf", most people will assume that it will behave at least roughly like a real-world creature, and the self-preservation instinct is one of the most basic traits of all living beings, so the "AI wolf" not having that is indeed surprising for a layperson.

"I really hate when people describe an ai as something that cannot be understood because they personally don't understand it."

On the other hand, keep in mind that a significant weakness of most modern AI research is that it's extremely difficult to understand: you have the input, the output, and a bag of statistical weights. In the story, you know the (trivially bad) function that is being optimized; in general you may not. It's not without implications for other systems.

Further,

"At the end of the day, student and teacher concluded two things:

"* The initial bizarre wolf behavior was simply the result of ‘absolute and unfeeling rationality’ exhibited by AI systems.

"* It’s hard to predict what conditions matter and what doesn’t to a neural network."

The tooling for understanding complex models is a lot better than what most people assume.

> The initial bizarre wolf behavior was simply the result of ‘absolute and unfeeling rationality’ exhibited by AI systems.

This is a bad quote. They should not say this. It's a poorly trained agent doing a decent job of a poorly defined environment. Absolute rationality conjures images of some greater thinking but its actually a really stupid model that hit a local maxima. Calling it unfeeling implies the model has some concept of "wolf" and "suicide" but it does not. Replace the visuals with single pixel dots if you want an honest depiction of the room for feelings.

> It’s hard to predict what conditions matter and what doesn’t to a neural network."

This is generally true, but it isn't true here.

If we play the analogy further: life is suffering, apart from the brief ecstasy of eating sheep. The AI was trying not to suffer, thus chose the boulder.

Did my best to translate the (misguided) fitness function to fiction.

Man is born crying, and when he's cried enough, he dies. -Kyoami in Ran.

Cutting one's losses early may appear to be the most rational act if trying to minimize an agent's total suffering.

Which is why some forms of Buddhism are basically a cult of death: https://en.wikipedia.org/wiki/Sokushinbutsu

> In the video game The Legend of Zelda: Breath of the Wild, the monks in the Ancient Shrines seem to be based on sokushinbutsu.

Factoid of the day for sure

David Benatar reached a similar philosophic conclusion due to his utilitarian views, which was amusingly put (with a sort of AI present, no less) in this webcomic: https://existentialcomics.com/comic/253

Thanks. I think I just found a new comic to read.

It's good because most people can understand it. I'd say it's a perfect strategy for a game, but if they're using evolutionary algorithms they should require some form of reproduction for the wolves to carry on. That would make the suicide strategy fail to propagate well. I can also see a number of possible strange outcomes even then.

What mechanism are you thinking of? One in which having offspring is rewarding and so enters into the same learning algorithm, or one in which the learning algorithm/action selection is evolved and differentially conserved?

You're conflating the evolution of the strategy with the idea of the evolution of the actor being controlled by the agent. To give an obvious example, if dying gave 100 points instead of subtracting 10, even the dumbest evolutionary algo would learn to commit suicide asap. The survival of the actor has no intrinsic relevance to how the evolution develops.

>> You're conflating the evolution of the strategy with the idea of the evolution of the actor being controlled by the agent.

Yes.

The survival of the actor has no intrinsic relevance to how the evolution develops.

No, not in this case. That was my point. That's why the outcome should not be surprising.

If I remember correctly there were similar scenarios that would occur using that popular Berkeley Pacman universe where he would run into a ghost to avoid the penalty of living for too long.

It reminds me of the thread about the Quake 3 bots, who left alone for several years, figured out that the best approach was to not kill each other.

https://i.imgur.com/dx7sVXj.jpg

Without knowledge of their reward function its difficult to tell if they're converged on this strategy or if its just broken.

The example you're thinking of is actually in gridworld [1]. As you allud to, one of the parameters of the model is the cost of simply being alive for an additional time-step. If the cost is negative (a reward), then the agent will just sit there forever and accumulate infinite points. If it is zero, it might still just sit there to avoid falling into the hole, which has a large penalty and ends the simulation. As you turn up the dial on the cost of living, the agent starts using more and more aggressive strategies to reach the goal quickly. But if you make it too big, it will just jump in the hole.

[1] https://inst.eecs.berkeley.edu/~cs188/fa18/assets/slides/lec...

Reply on news.ycombinator.com