I've had a chance to play with this and it's eerie how good it is. Most impressive to me, it was able to defend itself persuasively when confronted with evidence of how its previous answers were inconsistent (in the version I played with, it tends to be very coherent but frequently wrong).
I tried to get it to say controversial/inflammatory/bigoted things with leading questions, or questions where the framing itself was problematic, and it was quite adept at deflecting/dodging/resisting. I was very surprised.
I'm joking, I know that if it did say controversial things the headline would be "Google has created a robot fascist". You can't win at the moment....
What?? "You can't win" because this language model tries to avoid saying bigoted things? You say that like it's some sort of catch-22. It's a very desirable property and is a hard problem that needs to be solved to safely deploy these sorts of models to billions of users.
Despite changes in culture, people's nature hasn't changed in 200 years.
I've noticed contradictory statements are kind of standard for GPT-3. I can well believe you could come up with a system that's better at defending itself against the charge of contradiction than it is at avoiding contradictions in the first place.
I think it's a common weakness of chatbots that you can get them to contradict themselves through presenting new evidence or assertions, or even just through how you frame a question. I found LaMDA to be much more resistant to this than I anticipated, when I tried it out (I'm a Googler).
It wasn't completely immune -- I was eventually able to get it to say that, indeed, Protoss is OP -- but it took a long time.
Discloser: xoogler here