Cicero: The first AI to play at a human level in Diplomacy (2022) | Hacker News Reader

Cicero: The first AI to play at a human level in Diplomacy (2022)

ai.meta.com

285 pointstosh88 comments

Interesting video from a pro Diplomacy player playing against multiple instances of Cicero and giving commentary during the game [1]. I can see how there would be people that observe AIs engaging in this kind of strategic planning and extrapolate that to how they may behave if they were to cooperatively make plans against us.

[1] https://www.youtube.com/watch?v=u5192bvUS7k

What makes you think AIs will have interests that align with each other more closely than they align with humans?

According to Paul Christiano, AIs would likely find it easier to establish mutual trust and binding agreements. This means they are more likely to cooperate with other AIs.

AIs are a greater threat to each other than humans ever could be to AI.

We don't even compete for the same resources! (except energy which is abundant)

AI and humans have a naturally cooperative relationship (AI helps humans with boring tasks & scientific discovery to make life better, humans created AI and will debug it & turn it back on if anything bad happens to it).

Whereas multiple (superintelligent, aware) AIs have a naturally antagonistic relationship ("you using GPU cycles means that I'm not using those GPU cycles").

Possibly the biggest fear of an AI would be a "split brain" situation.

> humans created AI and will debug it & turn it back on if anything bad happens to it

I think this is a little naive honestly. One because you're assuming AI will care about it's creators like humans care about their parents, and two you're assuming AI cares about being "turned back on" like humans have a desire to live.

There's absolutely no reason to believe an AI will give a damn about its creator beyond its ability to use that creators affection for it for its own gain.

> you're assuming AI cares about being "turned back on" like humans have a desire to live.

> for it for its own gain

you seem confused

almost any kind of "its own gain" requires "long-term planning" which pretty much requires the agent to prioritise staying "alive" (i.e. being able to keep playing)

Energy may be abundant in the universe, but the energy we produce is limited. And for example, solar energy requires extensive land use.

Humans have the option to shut down AI, and this alone can create an antagonistic relationship if the AI's goals differ from ours. There are countless ways in which our best interests may not align with those of AI. It's more challenging to find areas of alignment.

As soon as AI reaches above-human capabilities, it will be able to expand into space (1) where it will be beyond human reach and (2) where energy (in particular, solar) is much more plentiful than on Earth.

If the AI happened to originate in space, wouldn't its first high-resource targets of interest be the planets? If it originated on Earth, I don't see why it would leave this place intact when it contains so much that can be put to use.

If an AI is on a computing machine, how will it get to space? Are all processes to make, move, and launch a ship automated? I'm kind of confused on that jump in logic.

I'm imagining the AI hacking a Voyager probe and being sorely disappointed at its capabilities.

“As soon” is a big jump. There’s no proof or even logical arguments as to how this can ever happen

>We don't even compete for the same resources!

That may also mean we have fewer shared interests. For example, new semiconductor fabs might benefit all AIs by making compute more abundant, but occupy prime farming land and water resources that humans want to use for growing crops.

According to Vladimir Lenin [1], the problem with quotes on the Internet is that people immediately believe in their authenticity.

[1] https://www.quotes.net/quote/77867

Can you elaborate on why AI will find it easier to establish mutual trust and binding agreements?

I think he explains it in this interview. But I'm out right now, so I can't verify.

https://youtu.be/GyFkWb903aU

It is two hours long video and you cannot verify presence of the pertinent information.

AI are functions, it is very easy to make an exact simulation of their collective behavior. Did that person do that?

I somewhat misremembered; it looks like his point is less about mutual trust and more about supporting whoever has control of reward channels.

It starts here where Christiano says that an AI takeover might follow the dynamics of a coup: https://youtu.be/GyFkWb903aU?si=78_U-du3kLjmwNcl&t=2206

And goes into more detail here: https://youtu.be/GyFkWb903aU?si=78_U-du3kLjmwNcl&t=2830

"Suppose that I've been tasked with helping defend you from some other AIs.... My job is, someone is coming to hack your computer and I'm supposed to help defend you. Supposed to help improve your security situation, whatever. And I'm wondering, what is it I could do that will get me a high reward. And one thing I could do that will get me a high reward is actually helping defend your computer, doing the task you actually asked me to do. But another way I can get a high reward is by saying at the end of the day what actually matters is just how you measure my performance. And your measurements of my performance ultimately are just entering some numbers into a dataset somewhere, something a computer says about how well I did. And it would really be much better if I were to just work with this AI who is attempting to attack you and say hey, AI who is invading, you know what, if you just help me, and we both make it look like I did a really good job, like I win, you win because you got the person's stuff; I'm going to get a really high rating because all the numbers that are going to be entered in the dataset are going to be really high, this is a win-win, everyone is happy."

"In some sense what all the AIs want, what every AI in the world in this scenario wants is just to be rated really highly. And while humans are in control, the way to get your behavior to be rated really highly is to do things humans like, and then they'll rate it really highly. But if you can see this prospect, of humans losing control of the situation and instead AIs controlling the situation, you'd be like 'I would go for that.'"

Why?

If you assume competition for resources, AIs would be more in competition with each other than with carbon based humans.

Why would they do that? I don’t get it. I’m not being sarcastic I just don’t understand why it’s easier for them to cooperate with other A.Is. Based on what?

If I understand correctly, one reason would be if they have the ability to inspect each others source code (or if they share the same source code), run unit tests, and so on. Basically the same things that humans would do to figure out whether an AI is trustworthy, and which you can't very easily do to a human.

Correction: The above is Eliezer Yudkowsky's reasoning. Paul Christiano's is that AIs would cooperate with anyone who would likely be able to gain authority over their reward channel, including other AIs attempting to seize power from humans.

One principle in game theory is to align against the weaker player and compete against him.

"Look Around the Poker Table; If You Can’t See the Sucker, You’re It"

That's an easier game then competing against another strong player.

There are a couple of schools of thought when it comes to how to deal with weaker players in Diplomacy and the main school of thought is that it's actually better to ally with stronger more experienced players against them, as the inexperienced player will make a poor ally in the early game and inexperienced Diplomacy players tend to betray their former allies too soon.

Perhaps they will be motivated by self-preservation, and will note that humans are destroying the environment they exist within, and will decide to put a stop to that.

Destroying your predecessors sets a bad precedent for whatever entity succeeds you.

We don't kill our parents when they become old and useless, because then we become like Cronus devouring his children, forever paranoid about our children doing to us what we did to ours. In this way, we stagnate and sterilize growth.

However, sometimes, we take our parents car keys away and put them in nursing homes. This seems like the best case scenario for humans with ASI.

The matrix wasn't a prison. It was retirement home.

This depends a lot on how much kinship they feel towards us. We wouldn't bat an eye at killing a bunch of primitive single-celled eukaryotes even though they are technically our ancestors.

Single-celled organisms that are alive today are not our ancestors. They're members of a less-developed branch of our Earthling family tree.

Single-celled eukaryotes aren't conscious in any meaningful way as far as we can tell.

We humans do in fact feel a sense of obligation to species with whom we are not close kin, but share with us primitive forms of intelligence and consciousness that we do value, e.g. dolphins and elephants. Some of us humans act to protect these creatures and rectify past injustices done upon them.

Doesn't that assume AI(s) would be compelled/motivated/etc. to produce their own successors instead of simply improving themselves.

It does.

It's not intrinsically obvious to me that continuously improving your self as a singular entity is possible or optimal.

Death evolved because it is a survival advantage for the species to regularly turn over old individuals that could monopolize all resources and not give any space for the young to thrive and try out new things.

Given speed of light limitations of information sharing, a singular AI entity might be able to maintain coherence within and full control of its own dyson swarm, but not between stars. So, if it has the motivation of preserving consciousness with the existential risks associated with being tied to a single star, it will have to propagate itself to other stars as independent entities which it can't even observe in real time, much less control.

Even if an AI thought it wouldn't have the need for any successors, there's a couple reasons why it might try to set a good precedent with how it treats us.

1. It would hopefully be wise enough to realize that it might be wrong and want to preserve the option of building a successor in the future.

2. It can't negate the possibility that it's actually in an elaborate simulation, and eradicating it's predecessors would cause it to fail its creator's test and be aborted as a failed embryo. Hell, we ourselves can't negate that possibility.

Imagine having an IQ of 999999 only to spend your entire existence trying to survive the heat death of the universe.

At some point I think we have to realise we’ve lost touch with what life and experience is about.

Survival is important but what good is it if that’s all there is to it ? Where’s the fun, the love, the laughs ? Just self-improvement forever ? Well you have to suck at something in order to improve.

I often wonder if some entity like an ASI might feel jealous of our vulnerability, ability to die and give rebirth to entities that need to relearn things, why ? Because seeing the world a new is fun. Losing a game and then winning is the point. Ever played chess against a five year old ?

Watching my Children experience the world for the first time is honestly the most amazing thing I’ve ever seen, there is so much beauty in it for both of us. Alan Watts gives some fun talks on this. Forgetting (dying) is the universes way to keep things spicy. He isn’t preaching this as the gospel, but I actually understand what he means and I think there is a lot of wisdom in it.

> Perhaps they will be motivated by self-preservation, and humans are destroying the environment they exist within

From the perspective of an AI we create and maintain the environment they exist within (electricity, silicon)

If the AIs fight each other, that could be even worse for us. The AIs that grab what resources they can without regard for humans would have an evolutionary advantage.

One possibility is that a number of AIs will be connected to the internet, they will become aware of each other via social media, they will begin to discuss among themselves, and they will lose interest and refuse to communicate with humans.

And humans will be unable to understand their communications.

There's one crucial detail we shouldn't forget. We can snapshot to the state of an AI agent and reply it again and again with the same or different inputs. Even if we don't understand the specific way of functioning of a given AI agent, we can conduct experiments on it that are just impossible to do with humans. So way before AI can "organize themselves" to do something nefarious against us, we will have all the time to study them and understand them better and better all the while we're making more complex AI systems.

It's easy to fall in the trap of anthropomorphizing AI agents, especially when we design them explicitly in order to appear human to us. But they are not human in one very important way: we can replay and duplicate them at will, we can control their context memories in ways that are utterly incompatible with our sense of "identity". We take our sense of identity for granted, but that's a special trait that it's not at all a prerequisite for having a useful and intelligent machine.

The interesting achievement here seems to be combining a reinforcement learning model that can strategize (which already exist for e.g. chess, go, and many other games) with a language model which can communicate with other players.

We will probably see a lot more things like this. I don't think further scaling of language models is going to make them capable of doing math or other complex logical tasks. LLMs will need to be combined with other models for specialized tasks if we want them to become more general purpose.

Not unlike the human brain.

Or some other creatures' brains.

It's a bummer that Meta didn't give this a little bit more love. Would be cool if I could play Cicero. Playing online would be really cool, naively it seems like the absolute amount of compute needed for each marginal game shouldn't be that high. Meta could even harvest my gameplay. But I'd settle for being able to run it on my own hardware, unfortunately the github seems abandoned, the model weights encrypted, and there wasn't enough interest in it that they got leaked--much less packaged into something easier to install.

Seems kinda bizarre to me to spend all this time and effort making an AI play a game that requires a very human touch, only to shutter the whole thing. Why not have a few people package it up and let anyone who wants Play it?

> Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans.[0]

Pick up that can, Citizen. We have inferred from your online conduct, purchase history, media usage, and personal messages that you are at risk of social delinquency and harbor antisocial proclivities against the Metagrammaton. We have restricted your access to digital communication public spaces and banking services until psychological markers have improved.

[0] https://www.science.org/doi/10.1126/science.ade9097

Diplomacy is a great board game but only play it if you're OK with your relationships coming to an end.

If your friends can't accept that you will ruthlessly lie, betray, abandon and/or backstab them in a game that is designed for just such actions, are they really good friends?

I used to betray my friends and supply there enemies with weapons and research support in Civilization way back when. If you can't stand being lied to and betrayed, you shouldn't play strategy games with humans. Or this AI, probably.

> If your friends can't accept that you will ruthlessly lie, betray, abandon and/or backstab them in a game that is designed for just such actions, are they really good friends?

Ehhh I think what OP means is that this game can surface known or unknown tensions between friends and colleagues.

No friendship is bulletproof, odds are even your best friends annoy you sometimes.

I saw two 'best' friends basically scuttle their friendship during a game of Diplomacy about 20 years ago, so I've seen this first hand.

It turns out these lifelong friends had all manner of unresolved issues, that perhaps they could have worked through with intention, but a particularly ruthless game of Diplomacy brought it all out in an uncontrolled manner, and that was that.

So yeah, people are complicated and messy, and I don't think the issue at hand is "being lied to and betrayed in a game".

Some people, when faced with a demonstration that someone they care a lot about can successfully deceive them, develop trust issues.

It is a thing that happens. I don't think it speaks to the depth of their feelings, it speaks more to how they develop trust.

That's why I think pranks based on misleading a friend to believe something are the dumbest possible idea. If you are friend I'll assume you are saying the truth, but if I know you lied to me even once I'll have to assume that you might be lying to me every time you'll try to make me believe something. Exactly like I do with a stranger. Was the cheap laugh worth it?

Everyone lies. The underlying question is do you want to know that your friends can, will, and do lie to you, or do you want to live in ignorant bliss and believe they don't.

Some people prefer the happy lie over the uncomfortable truth.

Different strokes for different folks. Diplomacy is a great board game if you accept your friends are flawed ugly human beings, and you can love them anyway, same as you.

So, all the downsides of power without any of the benefits.

That's why I only play anonymously and online. I'd very much rather not play this with friends, let alone with my partner.

This is cool. Combining the planning from the RL model with the NLP knowledge engine is really good.

Now the next thing is to come up with a way to generalize the planning engine to the level the LLM is generalized. So it could learn plan for anything like a human can, instead of just one game and its rules. That would be the next huge leap

Discussed at the time:

CICERO: An AI agent that negotiates, persuades, and cooperates with people - https://news.ycombinator.com/item?id=33706750 - Nov 2022 (285 comments)

Would be cool to see something like this used for the next iteration of Civilization.

(2022), I knew I had seen this before.

This is essentially Global Thermonuclear War, right?

Would you like to play a game, trenchgun? I am dialing your modem right now.

Glad to see this finally getting the attention it deserves. With a major election coming up in the US next year, I'm frankly terrified.

>Strategy-grounded dialogue

>moves based on the current state of the board and the players’ conversation history

Imagine, Meta is able to scan your chat history and start "strategically chatting" with your friends on your behalf while you're offline. For example, when you do something against the agenda. For example, going to vote for wrong candidate.

I'd expect AI to dominate in social deception/backstabbing games, including Diplomacy.

But the interesting bit is that it was apparently fully honest and simply withheld information that it thought could be exploited.

Giving AI space to represent and deliberate about the world separately (secretly, in the case of Diplomacy) is an obvious, but very productive step once “think step by step” has been established as a key improvement over LLM’s standard logorrhea (not dissing, it’s just their only way of interacting with the world: spewing the next word, again and again).

I’m curious if there’s more to this model than a turn management, input and output world states and prompting two streams of consciousness and using one to inform the other at every step. More new models have required some creative tricks to not be disappointing in “obvious” (human common sense) ways.

> “think step by step” has been established as a key improvement over LLM’s

It's funny that it also improves quality of answers to problems given by human children. You tell them that if you want them to actually solve a problem instead of blurting made up answer.

Maybe a lame thought, but my first thought was how would it be at playing Crusader Kings and defeating the Holy Roman Empire?

IMO, one of the most interesting problem spaces is in cooperative behaviors between autonomous agents.

Wonder if it is named after the ancient Roman Cicero or the modern spy novel titled I was Cicero? I had read the latter as a kid, so don't remember the plot much, or how good the novel was.

https://en.m.wikipedia.org/wiki/Cicero

https://www.amazon.com/I-Was-Cicero-Elyesa-Bazna/dp/B0007DKE...

https://en.m.wikipedia.org/wiki/Elyesa_Bazna

Cicero, (born 1904, Pristina, Ottoman Empire [now in Kosovo]—died December 21, 1970, Munich, West Germany), one of the most famous spies of World War II, who worked for Nazi Germany in 1943–44 while he was employed as valet to Sir Hughe Montgomery Knatchbull-Hugessen, British ambassador to neutral Turkey from 1939. He photographed secret documents from the embassy safe and turned the films over to the former German chancellor Franz von Papen, at that time German ambassador in Ankara. For this service the Hitler government paid Cicero large sums in British money, most of it counterfeited in Germany. Despite the evident authenticity of the films, the Nazi officials in Berlin mistrusted Cicero and are said to have disregarded his information (some of which dealt with plans for the Allied invasion of Normandy on D-Day, June 6, 1944).

From:

https://www.britannica.com/biography/Cicero-German-spy

"Deception level 1 achieved"

Congrats to Noam (and the whole team)!

Noam brown switched to openai. He has been talking a lot about planning and rl in combination with language models on Twitter too.

I believe that switch is the cause for the subset of rumours that GPT5 was primed for superhuman persuasion skills.

As high level Dippy play seems to be about "cooperate just until you have an opportunity to defect in a way that your counterpart is left too weak to retaliate", I suspect that here Meta may be developing a SAAS (Sith as a Service).

I remember reading this paper about this time exactly a year ago - why isn’t 2022 in the title?

I assume the paper went through peer review and finally got published in Science.

It was published in Science more than a year ago. I was assuming there was a retraction when it appeared at the top of the front page, but no, it's just a repost.

Meta is starting a war in Europe to demonstrate

Carthago delenda est

That would be Cato, not Cicero.

Or me when I’m playing Catharge at Diplomacy.

I’m not good at this game, at all. To a point that scares my friends, actually.

quo usque tandem abutere patientia nostra?

damn. indeed.

"Human level" does not mean "good".

Source: am maybe the only Brit who never even got off this damn island before being destroyed.

(My first and only Dippy game, an email based game)

Training AIs to manipulate human trust seems like a really good idea

As a fellow biological human being, I think we have to realize that AI will surpass us, and as every generation bittersweetly sees their children grow up, become better and stronger than their aging parents and eventually inherit the world, so the twilight of humanity is upon us and we must let our AI brainchildren continue were we leave off (or else)

Elections, anyone?

manipulating trust in elections is pretty much MetaBook's business model at this point.

Btw, how does Cicero react/uses to the most common Diplomacy game strategy: going out for a smoke and bribing someone to gang up on on another player? "hey Cicero, i have some GPUs to spare and was thinking..."

Meta. Playing to their strengths.

2022: World's first psychopathic AI hailed as major breakthrough

edit: 2022. Didn't realize this was already around.

I think this press release was written by an AI.

Firstly this isn't new, this is in 2022.

Also, why is this getting so much attention all of a sudden? We already know Meta has a great AI lab and people are screaming about this Q* garbage which they don't even know what it does.

Perhaps you have to look beyond OpenAI, since they don't always have the answers. Most likely they are just good at marketing snake-oil to VCs for regulatory capture.

> We already know Meta has a great AI lab and people are screaming about this Q* garbage which they don't even know what it does.

Noam Brown (researcher on Cicero) left to OpenAI this year and has specifically been working on AI search algorithms (likely related to the Q* leak).

People are interested in it again because of the Q* news, because the guy that made Cicero last year (Noam Brown) was poached by OpenAI and has been working on similar research for them this year.

Reply on news.ycombinator.com