CICERO: An AI agent that negotiates, persuades, and cooperates with people

On the other hand, the bot has no concept whatsoever of keeping its words. After saying words, it is free to change its mind about what moves to play, motivated from, for example, messages from other players.

It goes with the zeitgeist to argue for what makes the life of big tech companies hard, but they are big enough that they can afford things like that. It's smaller companies and academics that would end up not being able to innovate as much

Go down that road and you end up with an IRB evaluation requires for an A/B test that changes the color of a button

video: https://www.youtube.com/watch?v=efPrtcLdcdM

model: https://huggingface.co/ykilcher/gpt-4chan

Creating an AI to lie seems like the Wrong Path.

Zuckerberg should shut this down ASAP. If that's the case.

Similarly, in Go it seems unlikely that perfect play could overcome a nine-stone handicap (again, I could be wrong, I'm not remotely a dan-level player).

All to say, it seems likely that Diplomacy is a game where the difference between "the best human play" and "the best possible play" is much larger than either Go or Chess.

Diplomacy is a bit of a choose your own adventure game too. Like there's an objective criteria (average SCs at the agreed end of game) but the human tendency is to try and win individual games. Humans will often choose to play sub-optimal strategies for better entertainment value.

I think the real accomplishment here is the ability to fool humans into thinking their not playing a bot. That's an impressive thing to do even these days.

So far all this has been with one player, amid others, no collusion.

You’re going to be surrounded very shortly by sleeper bots, including on HN. Relying on dang and others to root out bots will be futile. A swarm could easily collude to downvote people or get them ostrasized by their own friend group, as we have already seen when it came to crypto, metoo, BLM, lockdowns, vaccines and now Ukraine.

It’s really not hard for a bot swarm to completely exploit society in these and many more ways, and we are not ready for it. By the time the botswarms arrive online, it’ll be too late to do anything.

Update: LinkedIn already has a huge problem of fake profiles applying for jobs and offering jobs. And Twitter is overrun with bots. But soon, bot-written articles will be out-shared by your own friends rather than that “hack liberal/establishment/hasbeen paper” NYTimes.com

The animatrix had a good sort of storyline on this, but it involved a lot of unrealistic violence

> ... by inferring players’ beliefs and intentions from its conversations and generating dialogue in pursuit of its plans.

As opposed to what? Not inferring intentions, or generating dialogue against its own plans, or at random?

It's just doing what a person does. It understands the other person, and says things to further its goals.

Incarnate technology deceiving humans is a different domain entirely. Human motivations are ultimately comprehensible by other humans.

But what do I know of the motivations of (for example) the Bible? What happens when the living incarnation of a holy text can literally speak for itself? Or when the average believer thinks it can?

Ultimately this may all amount to the same status quo, but seeing what cognitive distortions have come along with literacy, newspapers, radio, TV, and now the Internet, I have to continually ask, will I personally be able to maintain skepticism when the full brunt of an AI and its organs is suggesting faith otherwise?

You can call me an alarmist or melodramatic if you wish, but it should give everyone pause that the delusions of paranoid schizophrenics from the late 20th century are now basically indistinguishable from emerging popular technologies and their downstream effects.

1. Because humans have an understood limit on intelligence.

2. Because we have systems in place to keep humans in check.

3. Because humans have a distinct physical location and thus stricter limits on their direct and indirect influence.

4. Because humans can't copy themselves.

I can't attribute that the parent - I don't know the individual - but I expected a chorus of such responses. I feel like I can see them squirming in their seats ...

But I feel it too. There's societal pressure against doing good. It's often more socially comfortable to throw a cigarette butt on the ground than to pick one up and discard it. That's bizarre, it's perverse incentives, and we can and should change it.

Just to be clear, “lie” here means “outright lie”. From your linked article:

> I asked Goff about any major falsehoods or betrayals that helped him in his victories. He paused to think, then said in his soft-spoken way: “Well, there may have been a few deceptive omissions on my part but, no, I didn’t tell a single outright lie the entire tournament.”

I think the average person would still consider a “deceptive omission” to be a “lie”.

The problem with this statement is it assigns intention to a AI model. It does not 'intend' to lie... but still may effectively do so. Lying may be the wrong word (as it presumes intent)... it's hard to express the concern I have of a model learning from games like diplomacy without using words that infer intent. Maybe it is the idea of it learning to better manipulate humans.

But I would not trust a system, any system trained on diplomacy or any similar game.

That's not how I interpreted the paper. If I have it right, it chooses its message with its current most likely intent in mind, but it doesn't try to be truthful about that intent - it tries to generate messages a human might if they had that intent (so it might tend to be truthful to its intended ally and lie to the player it's about to stab). I don't completely follow the description of the message generation, though.

I would think you could avoid much of this issue by creating a more sophisticated structured game model and use the language model only for converting between structured and unstructured data.

Feel like a hack would have been to try to force dialogue into an extractable form that stored a state model relevant to the game, even additional hacks like asking the opposing player to restate their understanding of prior agreements; disclosure that I have no idea how the game Diplomacy works, so might be irrelevant.

Beyond that, no idea how Facebook manages its AI research, but quick Google confirms my memory that Meta/Facebook has done prior research on enabling AI memory capabilities related to recall, forgetting, etc.; which I mention just in case you were not aware.

I'm not arguing that one side outweighs the other -- adding up all the positives and negatives of nuclear technology would be a significant undertaking -- I'm simply saying that the researchers at Los Alamos didn't fully understand the ramifications of what they were creating, and it was potentially the case that the negatives would far outweigh the positive. One example is that so far we have managed to avoid global nuclear war. But no one could have predicted in 1945 what the likelihood of us avoiding it were.

Similarly with this technology: no one can predict whether this will be a mix of positive and negative, or overwhelmingly one or the other. That's true of most technologies, but to be clear, this is not the invention of Post-It Notes. It is absolutely the case that technology like this could fundamentally change the course of human history -- for good or bad.

And I'm not saying we should try to shove it back in the box. I'd be curious if that has ever worked. Nuclear, sort of? South Africa gave it up, after all. But to do that here would be futile and I'm confident actually push toward negative outcomes.

All I'm saying is that we should be cautious in our approach, not that we shouldn't proceed. In other words, I think "... we can't hold back as a species just because there are drawbacks. we need to build around them." -- we're in agreement :-)

Heh, there are two ways to take your point. So:

I hope most people are capable of discerning poor/invalid counterarguments and holding to their beliefs.

I hope most people are capable of being persuaded by valid counterarguments and changing their minds.

My point was that on the margins there are people who make the right choice for the wrong reasons: they are persuaded to a valid position by sub-standard arguments, or hold their valid position despite generally compelling arguments. And technology like this will (eventually) change that equation. Whoever the AI serves, their position, right or wrong, will be overweight in the public opinion because everyone gets exposed to the custom-made, specific-to-their-mental-susceptibilities arguments from sources they trust, because on other issues those sources agree with them. There are none so imprisoned as those who cannot see the cage. (not sure who said that first, can't be original with me)

I've often found that a particular search goes sideways for me, but I think (hope) that's because my standards have increased over time, not because Google has actually gotten worse. Or because there's simply more to choose from.

Interestingly, I still remember one of the searches I used with a friend to validate that Google was better than Alta Vista and Excite: to find out how the level of the ocean is different across the Panama Canal. No other search engine could answer that back in 2000(?). Today I search for:

different levels atlantic pacific Panama canal

And find plenty of information on the subject, regardless of whether I use Google, Bing, DuckDuckGo, or Yahoo!

Swisscows returns reasonable results, if fairly down the page.

Yandex gives poor results.

Ranking sites is expensive. You should refine your query to narrow down the possible candidates instead of going through more pages.

Diplomacy, however, you can't manage any attacks by yourself. You will either depend on someone helping you with an attack/defense, or you will have to trust that someone doesn't attack you as you overcommit. You don't just verbally agree to these things, either.

Every turn, players commit to writing what they are doing and all the moves are resolved at once. So a player can go from imagined victory to catastrophic betrayal in the reading of a single order. This means that someone was just lying to their face for 15 minutes.

Some people just can't handle that.

Oddly, being branded as a backstabber just entices others to try to use that backstabber against what they see as a common enemy.

Best game ever. (I only behave like this in game, I swear!)

1. It's a long game- after hours of play, getting doubled crossed unexpectedly sucks because of how much you have invested 2. The game almost purely relies on cooperation/other players. So double crossing someone really screws them- it almost certainly means I have no recource (luck, dice rolls, individual tactics). Getting double crossed feels like you got CRUSHED in such a complete way that I haven't felt in many other games

I have played a few times with a few social groups. Most were aware of the point of the game and enthusiastic going in, but even with that, people's feelings got hurt quite a bit.

I will say that Risk/Catan don't really cause the same feelings when we play. Diplomacy feels like a whole different level

The time that crossed the line: Poland wanted to declare war on Greece, no one else wanted to get involved on either side, and it was a pretty even match between them.

A friend not involved in this game was visiting family in the same city as the Greece-player, so we knew he'd be dropping by to visit him too. Poland-player told the "neutral" friend: if Greece-player shows you his civ game, take a picture of the screen when he's not looking and message it to Poland-player.

Well he did it, Poland declared war, and his knowledge of the position of troops lead to him winning the war.

"How did you know I didn't have any troops in my southern cities?" And Poland told him how he knew. It didn't end the friendship but there was about a month of not talking to any of us. When he cooled down there was a long meeting over whether that was cheating or "All is fair in Love and Civ". We could never come to a real agreement, 3 in favor of cheating, 3 in favor of dastardly but legal. But we now have an explicit rule of no screen sharing of any kind.

1. Length. An in-person game can last easily 12 hours, and is mentally exhausting. Getting 10 hours in and THEN getting screwed by your friend feels worse than a game with less time investment.

2. Design. It's difficult to survive the opening few hours without alliances, but only one player can win, so everyone is incentivized to defect at exactly the moment they think they can make greater gains by defecting than by cooperating. Being betrayed, even if you survive, forces a total rethink of strategy and ruins the next hour or two of gameplay.

[EDIT] Risk and Catan also have what are sometimes regarded as straight-up design flaws that can cause "zombie players" who stick around for most/all of the game but, very early on, no longer have any chance of winning. In both cases pretty much the only entertaining thing left to do is to be a prick to other players and try to play "kingmaker" by causing the second-place player to overtake the first-place one. Often, in Catan, if you're that badly screwed you don't really even have a way to do that much. This leaves the zombie player having a pretty bad time just about no matter what, and can leave other players upset if the zombie player stopped playing to win and started playing just to mess with other players. Risk, especially played in the (really, really not great, as much as I liked it when I was 10 years old) world-conquest mode, also has a problem with very long games and early player elimination, in addition to sometimes generating hopelessly-screwed-but-not-technically-out-yet players.

- Unlike with Risk, Catan or Monopoly, if you lose a game of Diplomacy you can’t blame bad luck, as there is zero luck involved. The only ones you can blame are the other players and yourself.

- because it’s multi-player, you can easily get beaten by players that, in your opinion, played weaker than you (“I was doing great until they decided to all go against me”)

- There’s no way to really play the game without investing serious attention.

Let me try: The games' mechanic mean that everyone is just fighting for territory, those who intelligently cooperate will defeat others. So in this situation, the most effective strategy is cooperating intelligently early in the game and betraying your allies mid-game. The insidious thing about the situation is that if you're playing among friends, the natural way to cooperate to leverage the trust you already have with your friends. And so what any winner does is be themselves, use the rapport they have with their to get cooperation but be just a little less trustworthy than their friends expect. That is where the tendency to break friendships appears.

That said, when I have played such games, because my group wanted to play them, I've had a policy of just not lying or making promises. I may say things like "that doesn't seem like it would benefit me does it, because then B will surely just toss me out of the lifeboat next turn", and of course NOT speak up when someone makes a wrong assumption to my benefit. It feels like I've won more than my fair share of those games still.

This creates a lot of opportunities for kingmaking and spiteful plays from people who are not able to win, but are able to make sure that you lose. And the worst part is that you're often forced into these situations through no fault of your own.

Diplomacy allows for all of the same plays, but as the player, you have way more agency about both getting into, and getting out of them. It makes the adversarial alliance-and-dealmaking part first and foremost.

I guess you never play poker because you don't like to bluff. Or you've never faked a pass in basketball before taking a shot. Or you and the catcher like to announce when you're going to throw a curveball.

Folks who play like that are game-group poison. Hell, they can even make RPGs a lot less fun with that crap.

1. A second place player has fallen behind in a low or no hard betrayal alliance and is no longer capable of a meaningful backstab, and has decided they don’t plan to backstab their partner because through their ongoing cooperation they’re the second biggest player and you’ve had a good game together. They work to cement your win, because picking the winner is often as fun as winning is.

2. Two main players and their side henchmen who are no longer serious contenders are forced into teaming to prevent the other side of the board’s leader alliance from running away with the win. There’s been a massive amount of betrayals and the table is about to have the crucial fight that will collapse one or the others’ line in defense. One disgruntled player who is on the dividing line of both alliances picks the winning coalition by lashing out against the closest player that screwed them over hardest, ruining that side’s coordination. The winning coalition breaks through, then the biggest coalition’s leader backstabs and eats its subordinates for the win.

In both cases, honesty and cooperation primarily decide the winner - either because other players have deemed you “deserving of the win” or “designated winner by dint of having successfully avoided leaving one or more key players disgruntled enough to tip against you”

You _always_ need cooperation in these wins, but you don’t always per se need to backstab people to win. Insofar as you do, those backstabs come in many flavors and often don’t feel stabby, stuff like “I’m just consistently benefiting a little more from our mutual arrangement than you are” or “I have no plan to personally screw you over, but I’m pretty sure Gary is going to do it for me and I’m not gonna stop him.”