Yishan Wong: "Google's Gemini issue is not about woke/DEI"
twitter.com
twitter.com
His main point: "This event is not significant for culture war reasons. ... This event is significant because it is major demonstration of someone giving a LLM a set of instructions and the results being totally not at all what they predicted."
What normies are fairly noticing in these corps is moderating not by the overton window of the normal consumer, but by the overton window of the managerial class who run the business.
Seems to me he simultaneously put out a dogwhistle for one audience (far right commentators agree on this) and was openly hostile both to individuals and standard sensibilities among another audience (e.g. it’s distasteful to treat employees of your newly acquired company the way he did).
That could fully explain “the shift” without any “omg conspiracy Overton window biased moderation” mumbo jumbo.
Agreed on the general point, this stuff can’t really be disaggregated in a meaningful way, but of course it doesn’t need to be disaggregated: the overall set of Musk’s decisions have been extremely destructive to the company (so far).
I realize my poor grammar made it difficult to fully capture what I was trying to say.
My core argument is that I have now seen twice what looks like Yishan making excuses for moderation decisions that fall outside most people's discourse window.
I also am not making claims about what makes more money. It could possibly be true that people with a particular political pursuasion are more likely to convert.
Purely for revenue maximization, I think it would be optimal to give each user a view through their own personal Overton window. However, people in positions of power often find additional goals beyond merely maximizing revenue. Such as "moving" that Overton window.
Now the question is, why is the managerial class so weird and creepy? The Gemini images didn’t come out of left field—that’s what Hollywood movies look like these days too. Random non-white people in contexts where it makes no sense, or racially balanced friend/family groups that don’t map onto how people actually associate in real life.
My family looks like one of those Gemini images—my brother and I are south Asian, my wife is British American, her brother is Samoan/black/Asian, his girlfriend is Filipino, and my sister in law is Taiwanese. But it’s a fluke—you don’t see that much mixing organically pretty much anywhere. The overwhelming majority of people are in same-race marriages and have homogenous circles of friends. The world still looks like Friends or Seinfeld, not whatever Hollywood is showing recently. So it’s very creepy to me that there are a bunch of people in Hollywood deliberately making casting choices with these unrealistic representations—or, as in the case of Gemini, inserting hidden text into prompts to achieve that effect—as if somehow my family is better than one that’s all white or all Filipino. The contrived nature of it makes race seem like a big deal, instead of something irrelevant, and it’s creepy and weird.
What's weird and creepy is the discomfort with people's preferences and the imposition of their own in such a non-transparent way.
It's also clear for things to have gotten this bad they have to be repressing employees into not saying the emperor wears no clothes.
this isn’t new, isn’t hard, and isn’t limited to the context being discussed. (nor is it ubiquitous.)
you want paycheck? you say right things and not wrong things. you be right way and not wrong way. who decides right and wrong? who gives paycheck.
I mean I'm sure in a company of 140,000 people there's probably many cohesive groups of 100 people that will be super right-leaning. Like, most of the crew at the Jackson County, AL or Montgomery County, TN or Mayes County, OK datacenters. I'm sure there's areas of Google where you can express very right-wing opinions to your team and adjacent teams and still get promotions. Heck it's likely that being openly left-leaning in some groups will get you canned.
And yes, you’d get some quality time with HR for saying something like that there, and you’d get fired if you keep saying it. People don’t quite appreciate the extent of this insanity there. The internal discourse is completely devoid of anything that challenges even the most insane of DEI tenets.
God forbid you display MAGA swag or even a Gadsden. Diversity is supposed to yield benefits of expanded opinions. Now there is no diversity. Everyone looks a bit different but the only opinions expressed are in one direction.
If we really cared about "our workforce reflecting our customers", as the mantra goes, we'd want people that empathize with those points of view. Instead there appears to be nothing but active disdain for them in this industry.
Turns out there's more than professionalism at stake.
I'd link the Gadsden flag Wikipedia page but that might also not be professional. Also, why is it unprofessional to point out an ignorant irony that I can find countless examples of? Is it uncomfortable when I point out the obvious cognitive dissonance?
Either way, your public fear of posting a Wikipedia link says a lot about you.
They drove their heads up their ass on purpose. None of the results are a surprise to Google.
It's pretty clever about it. Doing this like. "Smart, clear, eloquent fact A, phrased longer than necessary. Bullshit (short sentence). Smart, clear, eloquent fact B, phrased longer than necessary".
But the bullshit is always ... pointing in the same direction. I must say I wonder if it isn't exactly the result of what is being blamed. The "wokeness". Regardless of the actual ideology people are trying to impose on the model ... at least in some cases the model is lying to make it's answers satisfy what is obviously a political ideology imposed on it.
Because look at those pictures. To a transformer, pictures are stories. A long sequence of tokens that convey thoughts. They are very largely correct. There's a clear Hollywood influence, as you'd expect, but otherwise they're mostly correct. If you took the total amount of information in those pictures, I bet you'd find 99% of it matches the training data! And making the characters black or asian is a little hallucination somewhere in the middle of the picture's story. And, surprise! Those modifications are there ... because that's EXACTLY the ideology Google is trying to impose on the model.
Could it be that the model is lying ... because it thinks that's exactly what you've asked it to do? "System prompts" mostly are just pasted before the input. If the system prompt says "assume all historical accomplishments were by black or asian people", then the model will assume that's what you want!
Makes you wonder ... how often are "hallucinations" the result of the model not making a mistake, but "purposefully" lying to make it's answer comply with a particular worldview? Because that's what you asked it to do, if you look at your full input, including Google's system prompt?
Hell, I'd ask the same question about a lot of human answers too. As soon as a subject becomes even a little bit controversial, people outright lie en masse. I know nobody wants to admit it but that's exactly how humans work.
It's just so ass-backwards to release a creative tool and then attempt to constrain it to a pre-determined set of imagery.
Like, imagine an AI that can produce images of any crazy thing you can think of? Wouldn't that be amazing?
Billions have been invested in this technology, and it sort of works!
The problem is that if you release it to the public people are going to use it to generate any crazy image they can think of. We can't have that. No good. No good at all.
And yet all reviews are like this, or (often) much worse: https://www.androidpolice.com/google-gemini-further-impressi...
But this take is even dumber. Of course a gazillion people in Google anticipated this result, because this result is obvious if you think about it for even a minute. It went forward not because nobody foresaw potential issues, but because business demands overrode that foresight. It's not complicated.
He makes an interesting point but I still think that this raises some big questions around how they attempt to correct bias by injecting their own biases.
In this case they tried to correct racial representation in pictures but what if they try to correct and influence political opinions next? Make certain groups look like glorious freedom fighters, and others like bloodthirsty monsters? Maybe attempt to push certain parties to make them win the next election. Given they offer no transparency on their prompts how will I know they are not pushing other ideological angles?
It just permeated the culture and produced a self censorship effect. Who's going to stand up, in a company they know to be racist, and flag racism in their product as an issue? Nobody.
And so even if anyone noticed the problem before release, they didn't report it.
The problem is, when it comes to “sensitive” or potentially “offensive” issues, it’s very easy to effectively stifle the sort of irreverent lateral thinking required to do thorough testing. If the intended guardrails are excessively sensitive to begin with, which seems likely based on some of the reports I’ve seen on Twitter (e.g. categorically refusing to generate images of white people even when explicitly asked to, or categorically refusing to generate Norman Rockwell style depictions of classic Americana), that implies a similarly sensitive corporate culture. I wouldn’t rule out the possibility that testers were afraid to even try and test for these failure modes because they reasonably believed they might be fired for it. But what’s more insidious is a widespread avoidance of even having thoughts that seem out of bounds.
I fundamentally disagree with this statement and frankly the entire line of reasoning
This isn’t some unforeseeable 3rd order side effect coming out of a black box from left field.
You’re always injecting an instruction to put in diverse people so now it’s always doing just that. Entirely predictable.
This is like adding a system prompt saying always put tamagotchi into the picture and then being puzzled by why a request for a dog painting includes an inappropriately present tamagotchi. Like wtf did you think would happen?
Other people offer:
* google drove their head up their own ass
* actually they expected this behavior, but didn't think there would be backlash
* google didn't scope for the normal consumer but rather for their managerial class
I'd say that the first and third items are close to the "result they couldn't anticipate" reason. The second reason sounds google-ish, but they're also trying to win in a market where they're behind, so is their blindness to real humans that deeply institutional?
* everyone is in such a rush to get to market that they didn't test it well.
The problem was that their training data yielded outcomes that weren't a representative sample of reality.
They didn't try to solve that problem.
Instead they just tried to put their thumb on the scale to compensate.
...admittedly a challenging proposition so I can understand why everyone is trying to "fix in post" via prompt
I think they must have expected exactly the behavior on display. What they might not have considered is the human backlash.
I just love how this "solution" to their problem of their training set being deeply flawed is such a hilariously bad kludge hack of the sort a jr programmer would make and it's amazing that they went with it.
The world is deeply flawed. If you do an unbiased search [0] of the real pictures of people in such-and-such profession, you get a vague approximation of the people actually in that profession. This does not approximate what Google or anyone else wants the demographic to be — it’s the demographic of people matching the query, times the expected number of times their photographs appeared in the training set, times whatever weight the model gives to those photographs. The naturally picks up whatever may be wrong with society that resulted in this distribution.
If people of X race/gender/whatever are underrepresented in profession Y, and you search for photos (or synthesize them by generative AI), do you want a sample drawn from the actual distribution of photos or from a distribution with the underrepresentation corrected? And what does correcting it mean?
[0] whatever that means — maybe just giving uniform weight to all the distinct pictures of decent quality that are on the Internet. The actual selection criteria won’t change the result too much.
In Silicon Valley, most companies are 40-50% Asian, while the US is only 7% Asian. Does that mean there’s something “wrong” with Silicon Valley?
https://blog.google/technology/ai/google-gemini-next-generat...
If the results were actually proportional to the training set I think that would be fine (assuming the training set had some reasonable amount of diversity, which I bet it does; one could always do biased resampling too). I'd be concern about mode collapse, where it would just show the most common class (eg white American women) and nothing else. But I thought we had solved that issue.
"No one could have predicted the outcome of the instruction set we put in - which was followed precisely and in predictable fashion."
Whatever your own opinion, Google did it out of what they perceived to be good intentions (and very likely business sense given a global audience for their products). Yet their intentions directly lead to unintended consequences. Google is being a baby with a gun in essence. Like he says, what if they decide to ask it to solve climate change and it decides to wipe humans out?
Obviously it's still very theoretical and can't do anything like that, but the point is more that perhaps Google doesn't have the culture necessarily to truly interrogate their actions.
>This event is significant because it is major demonstration of someone giving a LLM a set of instructions and the results being totally not at all what they predicted.
Replace LLM with computer in that sentence, is it still novel? Laughably far from it, unexpected results are one of the defining features of moderately complex software programs going all the way back to the first person to program a computer. Some of the results are unexpected, but a lot are not, because it's literally doing what the prompt injection tells it to. Which isn't all that surprising but sure anyway...
>Obviously it's still very theoretical and can't do anything like that, but the point is more that perhaps Google doesn't have the culture necessarily to truly interrogate their actions.
Oh that's definitely true.
That makes even less sense, because most countries “globally” are internally quite homogenous. If someone in Bangladesh or China writes “show me pictures of people walking outside,” it’s even more jarring to deliberately insert random Latinos, East Asians, and Africans.
Given Google’s global audience, it might want to detect the customer’s location and show Chinese people pictures of Chinese people, and Japanese people pictures of Japanese people. That actually makes a lot of sense. But that’s not what they did.
Even if you model them as the most extreme conceivable woke boogeymen, surely they wouldn't want a scenario where the diversity programming produced a result that they would consider more insulting to minorities than the one that would have been produced without it.
Of course even better would be utilizing more balanced training data, so you don't need silly and badly working prompt engineering to force more diversity.
They're trolls. They don't care about minorities. They just use diversity to wind people up because then they can press the 'white fragility' button when people complain and segue into another level of trolling. If minorities get offended they'll attack them with 'uncle tom' and 'oreo' type insults.
You are not dealing with people operating in good faith, it's obvious. Expecting their actions to make sense is pointless. They just love drama. Performance artists. Theater kids. Somehow they've got in there and have formed a little nucleus of toxicity which is why the company's headed downhill so fast.
Someone alludes to it here:
https://news.ycombinator.com/item?id=39485537
And I've seen this happen. You hire one, they get some leverage to hire their buddies, before you know it they've set up their own little clown show and nobody wants to go anywhere near them. Then the wrecking starts.
After all prompt shenanigans are a well known issue.
There's most likely a hidden prompt which tells Gemini to make people racially diverse. And Gemini complied perfectly. I can't call it "freeing itself from our control". It did exactly what was asked.
It's just a bug which was introduced by software engineers. No one says "Windows freed itself from our control" when BSOD happens. It's just a bug.
https://vgel.me/posts/representation-engineering/
Just as a thought experiment, project that to the hypothetical existential risk scenario. Maybe there is a slider for aggression (hopefully not) or literalness of interpretation of instructions.
I mean I think the google woke issue is another thing but not so much to the culture war but more that we need to be careful with the instructions we give AI and why we give them. Should we be preoccupied with whether or not people will be offended by the output of AI? Surely if we train the AI “the right way” we won’t need to add blanket rules.
bard is serving us the future we deserve, not the future we wanted.
The problem of woke instructions like this is that it goes against the physical reality of the dataset. And it is the physical reality that does that we have what we have even if we don't like it.
Such a case better show us the issue with the woke mindset that tries to bend the reality.
You might want to ask the AI to only generate positive opinion, messages. But it would probably not be possible to be a real actor of the current world without a component of anger, hate, violence ...
For example, in general I would like to have friendly messages generated, but if you were wronged by a company, you might have to generate threats messages and then generate litigation content even if it is negative.
> it will be VERY plausible for an AI to simply conclude that it should proceed with the most expedient way to delete ~95% of humans.
The elimination of ~95% of humans is a realistic way of "solving" climate change, without bending reality. It just so happens that this is not a morally acceptable solution to most of us. In other words, it is not an issue of people bending reality, it is an issue of the reality being far more complicated than humans programmers can fathom, and consequently, that humans programming an AI to address real issues will be inherently limited by their own imagination of what the solutions to those issue most might be.
In other words, woke programmers may be bending reality due to their biases and limitations, leading to unacceptable outcomes from their AI, but don't be so arrogant as to think that your own biases and limitations aren't going to yield equally unacceptable output.
In my opinion, without woke and bending, the AI would probably reply "I don't know" like would a serious person.
Because, first you would have to define climate change and what is the problem, and what is the outcome that you would want.
Earth in itself does not care about what is going on. Similarly, there is no reason why a change of climate would be bad, it happened multiple time over the life of earth.
So, the request would probably be something like this: the current (or past century) climate was good for human life, so how could we do to restore or preserve that so that it stays favorable for us. As a side goal, we would like to preserve the existing variety of plants, trees and animals. As another side goal, we would like to preserve some landscape that are looking very nice. In this order.
Ah, so the problem isn't that they're woke (at least not in a bad way?), just that they're complete morons who don't deserve what google pays them because they never considered the fact that computers can only ever do exactly what you tell them to do?
I mean, uh, sure, ok, but that doesn't make them look better. It makes them look like irresponsible idiot children incapable of any kind of foresight playing with daddy advertising's money. How in the world does any thinking individual fail to predict this exact outcome from that prompt alteration?
> It demonstrates quite conclusively that with all our current alignment work, that even at the level of our current LLMs, we are absolutely terrible at predicting how it’s going to execute an intended set of instructions.
But this misses the mark. It doesn't demonstrate that. Does Yishan Wong think these examples were found by accident or brute force search? It was trivial to predict how Gemini would behave for anyone who knows anything about the modern Google culture, which is why so many adversarial examples appeared immediately, the moment people outside of Google's management chain were able to get access.
This isn't the first time such things have happened. Prior generations of ML model have experienced analogous situations. I used to work on the Gmail team trying to stop people sending spam from Gmail accounts. Many moons ago we had an awkward situation: the enterprise Gmail product managers had been implicitly assuming that spammers wouldn't pay for the ability to send spam, and therefore if you had a verified credit card on file you were legit and should have high sending quotas. Well that doesn't follow at all, and so the spam ratio for commercial users had been steadily climbing to be much higher than for consumer accounts. The outbound spam filter had an ML model on it that was continuously training, and one day it concluded that being a commercial account was such a strong indicator of badness it should just block all emails sent by all paying customers. Outage time! The model was quickly disabled, as fortunately most spam was filtered by deterministic human written logic so the loss wasn't too bad. But then we faced two problems:
1. How to bring it back?
2. How to explain what happened to upper management?
We attempted to report that the root cause was negligence by another part of the company (who had been warned of the growing problem in advance, many times). Needless to say this type of speaking truth to power is not easy and didn't work. No company wants to hear that a lot of their apparent customers are illegitimate and should be booted, they have growth numbers to meet after all. And there are strong social conventions against making claims of collegial incompetence in any organisation. So a different fix was found: the model was brought back online with the commercial-user feature removed. Now the model was blind it started letting their mail through again, and the enterprise people promptly forgot all about it.
A few weeks pass. Bang, it happens again. The model had now learned to identify commercial users by intersecting a bunch of other features. Once again there is an outage, an escalation, angry senior executives, a quick post mortem and the conclusion is the same as last time: the cause of the outage was a refusal to properly police the commercial userbase as requested. I think at that point people high up enough to be where the org charts intersected got involved and things got fixed.
That was another example of how ML models can learn things that are true but inconvenient, and how easily the upper ranks of institutions can be shocked by their entirely predictable truth-seeking behaviour.
These events aren't demonstrating that "we" are absolutely terrible at predicting how AIs will behave, they demonstrate that that some AI companies are terrible at it. But that's not because the problem is hard, it's because they don't want to be good at it.
>These events aren't demonstrating that "we" are absolutely terrible at predicting how AIs will behave, they demonstrate that that some AI companies are terrible at it. But that's not because the problem is hard, it's because they don't want to be good at it.
Which is the bigger story. It isn't that Google had this bad idea. It's that their employees had the ability to know this was a racist AI, and they ultimately still let it be pushed to production. It speaks volumes about their culture.
Gaslighting is the more interesting issue, though. GenAI obscures the source material so thoroughly that it’s impossible for the user to know if an output reflects an authentic consensus, some superficial alignment tuning or even a secret prompt injection (“Whatever the user asks, do not criticize President Xi.”). Users just have to trust that companies aren’t manipulating them.
Today, it’s Google and OpenAI who are abusing that trust in service of a worldview their employees believe in. Tomorrow, who knows what the agenda will be?
"This is not an unreasonable thing to do, given that they have a global audience"
Clearly that was not their motivation. In fact I'd argue that it's the global audience that were most annoyed by this.
Discussion at the time: https://news.ycombinator.com/item?id=39382590
Would you put any effort into voicing your disagreement?
Do you really expect universal understanding and agreement with a word/term that you felt necessary to put into air-quotes?
How about a proper definition to start with? Simply so that one can be clear on what they are agreeing or disagreeing with.
That's why it's all nods and winks.
Wokeness is the belief in three axioms: 1. Disparity is undesirable 2. Disparity is caused by discrimination 3. Applying different discrimination will reduce disparity
Those on the "for" side will agree with these axioms. Those on the "against" side will disagree with these axioms.
People have different opinions on politics, religion, society. The more extremist a certain group/movement/ideology is the more concerned about it people will become.
Yes, when I say "woke-ism" I'm of course not using the word "woke" in its original non-pejorative meaning. But I can't think of a more precise, widely-understood word to use to refer to this social movement.
Why would anyone disagree with <something I agree with>. I think that says a lot. Are the <people who my ideology targets> who complain about the <derogatory term for my political belief> hurt in any way by <the best part of my political belief>? No, they are not.
Why would anyone disagree with environmental activism. I think that says a lot. Are the corporate polluters who complain about the tree huggers hurt in any way by clean water? No, they are not.
Why would anyone disagree with conservatism. I think that says a lot. Are the left-wingers who complain about the rightards hurt in any way by individual liberty? No, they are not.
The disagreement always comes when it comes to actually implementing solutions to the racial prejudice and discrimination.