Google CEO Sundar Pichai says its malfunctioning Gemini AI is 'unacceptable' | Hacker News Reader

Google CEO Sundar Pichai says its malfunctioning Gemini AI is 'unacceptable'

gizmodo.com

59 pointsantimora78 comments

Is it malfunctioning? Or is it just being generative AI? Maybe it's time people accept that generative AI is good at making shit up, and so we should just use it for that. Maybe we should stop pretending that "probabilistically predicting the next in a series of words" counts as intelligence.

I say, let the generative AI be racist. Let it hallucinate and come up with all kinds of crazy shit. And with that, stop using it for things that matter. Use it for art, entertainment, experiments, explorations. Use it to mine the depths of the human soul and reflect back at us what we are. Don't use it for law, health care, directions, or anything humans depend on. Acknowledge that it's just a stupid program with output that "looks impressive", and that we've all been fleeced into thinking it would be anything more. Regardless how many billions we sink into it, it only appears intelligent when we conceal how batshit crazy it can be.

> it only appears intelligent when we conceal how batshit crazy it can be.

this can be applied to a broader aspect than just AI. in today's post-fact world, this is absolutely applicable as well to pretty much anything in life. we have people believing in the wildest ideas and not as a joke or just for the lulz, but wholeheartedly believing them as fact. people believe the earth is flat, that vaccines make you magnetic, or any of the other well known tropes.

all of that content has been used to train these "AI" systems. why would we ever expect it to not make shit up when a non-insignificant amount of its training data is made up shit.

thats not at all what is happening here. it looks a lot more like Gemini is following direct rules from a kind of DEI organization than anything else.

I agree with everything you wrote, it's the mindset I wish people would have.

One note, though, in this case, it wasn't the generative AI that was racist, it's the people who added the extra configuration of not allowing (or at the very least, strongly discouraging) white people in their images. The racism wasn't a bug (like in the past with the gorilla incident), this time it was a feature that was deliberately put there.

From a game theory perspective, Sundar's approach is actually probably the safest approach. If they didn't add the "over the top" diversity and inclusion filter, Gemini would probably not be given a second chance if the twitter-verse was able to generate anything remotely objectionable as presently the diversity and inclusion crowd has somewhat captured social media, etc. and is quite vocal and experienced w.r.t. canceling people/companies/things.

If Google adds the diversity and inclusion filters and the social media influencers/stock market want it to be less filtered, then Gemini gets to live another day. The Gemini team can go in retro-actively and remove the diversity and inclusion filters and Sundar can write an apology letter.

Then if the Gemini team goes too far in the opposite direction, there is precedence to apologize again as they fine tune it.

That is actually excellent PR. I would also add there was a more minor benefit from causing a reaction on social media in the first place, which indirectly advertised their product.

It is malfunctioning in the sense that it can be conditioned to not be like this. If you ask Midjourney to generate a picture of a Viking warrior, it doesn't spit out a central African guy wearing a Viking armor. It is not malfunctioning in the sense that it was conditioned to generate this kind of "2024 inclusive ad", where every group of people portrayed has an equal number of racial representative.

> Use it for art, entertainment, experiments, explorations. Use it to mine the depths of the human soul and reflect back at us what we are. Don't use it for law, health care, directions, or anything humans depend on.

you could, but the companies building these things very much want to sell it to other companies that do "things humans depend on", because that's a much larger market than just the entertainment industry.

Exactly. There’s no way the VCs who were dumb enough to invest in AI will capitulate so soon. It will happen eventually, but we have to put up with the baseless hype for a while longer.

>Don't use it for law, health care, directions, or anything humans depend on

This is kind of wishful thinking based on the idea that humans are much better. I've used it on law and health and ok it's not as reliable as a good professional but it will answer fairly well in a second or so whereas the professionals will be busy for a month, bill hundreds and you may still get incompetence. I'm also waiting two months for human solicitors to do a standard loan agreement, £2k in fees so far and no loan agreement yet. They are crying out to be replaced by AI as soon as it's a little better.

I think the biggest strategic oversight that Google leadership has made is not learning from Microsoft's Tay fiasco.

What Google should have done is setup an entirely separate startup (or startups) similar to the approach Microsoft has taken with OpenAI. Let a few young, ambitious startup CEOs lead the way and supply them with billions in funding which they must use in Google Cloud Compute AI spend - in exchange for a majority stake in their company and integrating with Google properties only.

Book the GCP AI revenue and get stock bump from YoY GCP growth. If the startup(s) misstep, claim the tax write-off then move on to the next batch. If they are successful over a five year stretch then just buy them out fully and get some positive news coverage over the successful strategic acquisition.

As a member of Mag7/FAANG, Google leadership must realize that any and all minor missteps are going to be blown out of proportion via clickbait articles.

[1] https://en.wikipedia.org/wiki/Tay_(chatbot)

Startups also provide brand isolation. If you get the Microsoft AI to say things it shouldn't say then that tarnishes the MS brand, if a MS associated startup has prompt hacks, then there is a layer of isolation.

Large companies are sadly not good at innovating themselves, probably this is an emergent effect from the size alone and the bureaucracy such size sadly requires. Especially when people sue you for millions when you search their name and lies show up, you must erect barriers. You also cannot let every PM in the company launch potentially disappointing products. Instead proposals need to get reviewed at least from legal points of view, but usually by a larger set of departments.

By the time half of the reviews are done, the competing startup has already launched the product, gotten direct user feedback, and established itself as the "innovator", all this while the big tech company where someone might have had the same idea is busy with getting it through its bureaucracy.

Agree - I do think that part of the bureaucracy is due to the fact that any new consumer facing product from a FAANG is going to be relentlessly challenged adversarially by every social media tech-influencer as in this case. And no product team wants to put the CEO in the media/congressional hearing hot seat.

Conversely a new consumer product from a new startup is going to spend half the time talking about what the new company is etc. before getting to any product flaws. And most of the time such things are relegated to niche reddit subgroups.

> probably this is an emergent effect from the size alone and the bureaucracy such size sadly requires.

One of the top complaints I've seen from former Google employees is that they have a ridiculous amount of bureaucracy involved in getting anything at all done. So I'd say there's a lot of credence to that assumption in this instance.

He talks about it as if it was someone else's fault.

His internal policies of extreme virtue signaling and DEI focus could only have led to that outcome. And now that I think about it, this was probably not a problem of lack of testing - it was tested and the results appeared as expected given Google's past of trying to mold society through subtle but deliberate biases in its products.

The issue is anyone with an IQ over 85 can tell that this behavior is in no way a "malfunction" of the system. The system was intended to operate this way and the general population found the output abhorrent and wrong.

The way Gemini talks about certain events (and generates images) is clearly in line with the walled garden of compromised liberal thinking at google. I say this as someone who votes blue and considers themselves progressive.

What's unacceptable, is Google clearly thinking it was okay to use their AI to erase history how it actually happened and just play it off as their org having moral highground.

The thing is that this got released to the public after _many_ people said it was ok, internally, at google.

You really have to lost any grip with reality to go for something like that, so I guess they'll rationalize some excuse. That will be interesting to see.

Having talked to friends who currently work at G in Mountain View - "lost any grip with reality" is an understatement when it comes to the collective ethos of G employees.

Looks like the "enlightened" google employees found this comment and applied "first principles" thinking ;)

If you read his full memo [1], it's the most Sundar Pichai response possible to a crisis. It's full of vague MBA speak ("red-teaming", really?) with the obligatory Google mission statement sprinkled in along with self congratulatory statements of their AI achievements. It does not inspire any confidence whatsoever.

[1] https://twitter.com/TechEmails/status/1762849036363505996

Red-teaming came from think tanks helping the military. It's a sensible term in this context, basically they act to anticipate the reaction of the public by entering offensive queries and reporting on them in a way that can override the blue team's shortsightedness.

What I really don't like is the "working around the clock". There is no need to rush a solution as quickly as possible, and will only serve to make the results worse because as we all know, googlers don't like to work around the clock and will not give the best results if they are forced to. Except if promotions are on the line.

The reason Google has about 100 DEI employees with their hands on the controls of this thing is because they believe anything else is unacceptable.

Gemini wasn't shocking because it showed racist and historically inaccurate material, it was shocking because it showed the world through Google's lens.

As others have pointed out, even Google search has the same issue. Image search will highlight minorities even when the search term suggests none should appear.

It was 100% deliberate, and they spent countless hours refining, polishing and testing the model. They were happy with it.

The funny thing is that for the images results it was hard to say if there was a manufactured result, an incidental one or completely nonexistent (ie. it's in people's minds). But in this case it's completely impossible to argue anything but an explicit instruction and comically explicit at that (by basically adding the desired result to the prompt). To be clear though this isn't a comment on weather search is being explicitly augmented in a similar way.

Which minorities are those, exactly?

Things like a black 1930s German soldier, for example. Google shoehorns in diversity so hard that it simply invents historical diversity in the name of DEI.

Google claims they were trying to make it reflect the global population, in which case those it tried to erase are the global minority.

I'm surprised their LLMs passed the strict Google launch review process, how do you test something that gives a random response every time you query it?

AFAIK they intentionally inject randomness to reduce the effectiveness of jailbreaking.

Their launch review process is anything but strict. Google makes awful launch decisions all the time. The culture is one of launching rather than launching something good, and there's no individual accountability for anything, as evidenced by Sundar's email.

[dupe]

Lots more discussion over here: https://news.ycombinator.com/item?id=39534608

Other than for entertainment purposes, who cares what a statistical analysis of English has to say about ethics?

In theory, nobody should, it would be ridiculous. The answer to your question is: lots of people today, probably even more tomorrow.

Well, perhaps we ought to educate people better. And quickly.

The real danger of LLMs is that people come to believe they hold some sort of authority — or that their output is in any way the end result of a process of deliberation and contemplation rather than simply being a (stochastically perturbed) best guess at what an answer should look like.

Yes, people already take Google at face value, but at least (in theory) they know that anything they’re finding there is written either by an organisation with an axe to grind or an individual no more omniscient than they.

If that's what it was, analyzing an entire language would be an interesting lens into how a large number of people think about ethics. But this one is more interesting because it shows us how Google thinks about ethics.

We really need something like The Onion with quotes from users, the way university newspapers will have a row of headshots and quotes from random students asked some question. The Onion of course was orginally a student newspaper that had something like that.

Instead all we get is an endless stream of BS from "Big Tech CEOs".

I want to hear what the people say, not some "Big Tech" compulsive liar.

Just thought I'd chime in to say this is often called a 'vox pop'.

The system was designed to do that. It is not an AI problem it is a product problem.

This could just be a real-world example of the difficulty of the alignment problem.

Having no personal insight into the minds of Google engineers who built this thing, my assumption is they wanted Gemini to give diverse results when someone asked for a "person" or an "accountant" (reasonable), but didn't think of cases like "English kings" (unreasonable) where the added context changes the distribution. So they added some one-line hack to the system prompt (we know this is how OpenAI achieves this) which unbiased the former distribution but added lots of bias the latter. Quite an easy oversight to make.

This is what AI risk people have been saying. You can't get AI to behave how you want exactly because it's extremely difficult to encode your values in a way where there's no unintended consequences.

And of course this was gang-flagged by the blue-haired fairies.

While everyone on HN is hyperventilating about DEI crap, the malfunction Pichai is most likely referencing is Gemini AI calling Modi a fascist. [0].

A Union Minister has already gave a veiled threat that Google broke Indian Telecom Laws [1] surrounding fake news.

It's election season in India and governments get very ban happy during this time (look at what happened to Amazon India in 2019)

Edit: Please don't make this an Indian politics flame war. The point is, a public company like Google is constrained by PR.

This is why OpenAI has been innovating so fast - they are private and don't have to answer to almost anybody

Public companies need to protect their brand, otherwise they lose out to similarly sized competitors (eg. Microsoft or Facebook)

[0] - https://www.thehindubusinessline.com/info-tech/googles-gemin...

[1] - https://twitter.com/Rajeev_GoI/status/1760910808773710038

The link you cited, says Gemini says: 'Modi has been "accused of implementing policies some experts have characterised as fascist"'.

Is this true? Or more specifically, is this issue here that Gemini is saying something that's accurate but that the Indian government doesn't like, or that no experts have characterised Modi as a Fascist and Gemini is saying something inaccurate?

It doesn't matter.

The issue is Gemini AI spewed an answer that caused Google to enter Indian culture wars less than 30 days before national elections.

Most of the horrible crap that happens in India happens during this time of year, because Parliamentary elections are very high stakes.

Most companies want to stay out of these kinds of culture wars, and Google inadvertently entered one

It's only a small step from rightwing populism to fascism. Rightwing populists say some things (like "we'll let Russia invade countries that don't spend 2% of their GDP on defence"), fascists actually do it...

It's only a small step from any extreme ideology to an even worse one

I guess my metric is, would you count AMLO as fascist? AMLO and Modi are functionally the same.

(Personally - I think Modi is an Authoritarian in the same mold as Indira Gandhi, Erdogan, Lula, Obran, and AMLO).

I think you're mixing up your populists here... AMLO (Andrés Manuel López Obrador) and Lula are left-wing (so you can maybe call them communists if you want to insult them, but not fascists), Orbán, Meloni, Erdogan, Putin, Modi, Trump, Bolsonaro etc. are right-wing/nationalist.

Nope. This is my point exactly.

All 3 are electoral authoritarians that came to power due to public anger at a corrupt status quo (Maluf in Brazil in the 90s-early 2000s, Coalgate/2Ggate/Jan Lokpal in India from 2011-13, and 2017 gas price hike in Mexico) and enabled by cheap populism and campaigning directly to the subaltern of their countries.

Furthermore, all 3 are products of dysfunctional democracies where criminalization of the electoral process has been the norm since the 1990s.

While electoral institutions Ann all 3 remain somewhat strong (as is reflected in the EIU's democracy index), civil society has died in India, Brazil, and Mexico.

Furthermore, all 3 leaders are notoriously anti-ideological in the sense that they will take right leaning and left leaning positions on policies at a whim depending on the electoral winds

How is a non generally intelligent AI like Gemini know now is a bad time to talk about Indian politics? Are models in general supposed to avoid offending anyone, anytime? Companies align the models so hard to avoid controversial opinions they lose performance. Is any kind of political question out? Is asking if Hitler was a Fascist also out?

> How is a non generally intelligent AI like Gemini know now is a bad time to talk about Indian politics

It can't and any attempt to prevent such an issue will only degrade it's capabilities.

> Are models in general supposed to avoid offending anyone, anytime? Companies align the models so hard to avoid controversial opinions they lose performance. Is any kind of political question out

I 100% agree with you.

And that's why this is such a shitshow.

Google has been so slow with rolling out LLM capabilities despite basically inventing them almost a decade ago - OpenAI doesn't have a $1.7T market cap to defend and Investor Relations to deal with.

I would say it’s reasonably accurate to say he’s accused of it given the number of articles and related information that pop up searching for Modi and Fascism - for example, https://en.wikipedia.org/wiki/Hindutva

I don’t know enough personally to make any kind of claim beyond that though.

To be honest, whenever I read about a politician that he/she is accused of implementing policies that experts have characterized as fascist", my only thought is "Who wasn't?"

It means nothing, and I hope more and more people realize that it's all just propaganda and low-effort, mealy-mouthed slander.

There's no problem if you just cite which experts and their credentials, even better if you can add the direct quote to remove all ambiguity in the first place.

That's expensive and enters licensing issues.

Corporate License LLM do give this option, but we are also paying an arm and a leg for a Glean, Azure GPT, etc.

FTA:

>“I want to address the recent issues with problematic text and image responses in the Gemini app (formerly Bard),” Pichai said in the memo

How does your interpretation fit into the 'image responses' part? That seems to be referencing US issues.

Notice how he mentioned "problematic text" first before images.

I've worked with PR/Comms teams a lot and this kind of edit was 100% done on purpose.

Also, note that Pichai and Raghavan went to college and grade achool with a number of these mid/upper level Indian politicians.

This is another symptom of the same issue, it's not like they're disconnected. They're able to ignore the elephant in the room and act like the biased output is an affront to people of color when their models depict black Nazis after being specifically aligned to erase white people, or when they express bias in non-US politics after embedding political bias in their training, because that's what's politically correct at this time (though I'm hopeful that we're on our way to some semblance of sanity as the fallout of DEI becomes more and more ridiculous and visible). Could you imagine any big tech CEO aside from Musk criticizing the model's political bias in a US context?

It is truly indicative of the shortcomings of the supposedly enlightened Hacker News hivemind that Google - a company that has killed influential projects and sacrificed / cannibalised the integrity of it's core search product - should be condemned because it's latest AI malfunction is too 'DEI'.

Google's AI blunders are more systemic than the latest culture war. The cringe hate spewed by earlier models are just as incompetent as Gemini, these are nothing more than the flailing of a multinational who is incapable of fashioning a vision of a world they promise. These are bugs at the highest level, that are clearly poisioning every layer of Google. They are the same bugs that led to Google search shortcuts that pushed disinformation COVID.

This is a failure of rigour from the flailing of the world's self appointed organiser of knowledge. Your least favourite DEI minority has nothing to do with it. For fucks sake. Place your blame where it belongs.

>supposedly enlightened

What made you suppose this?

>The cringe hate spewed by earlier models are just as incompetent as Gemini [...] Your least favourite DEI minority has nothing to do with it

Except unlike Tay ect. this would specifically add "of various diverse ethnicities" to each promt, so this isn't just some "bug" but an intentional decision. Do you seriously believe Google's DEI initiative is unrelated? Why ignore what they specifically say they are doing?

eg.

>moving forward, all VP+ performance reviews will include an evaluation of leadership in support of diversity, equity and inclusion. [0]

[0] https://about.google/commitments/racialequity/#our-commitmen...

Do you really think that a DEI mandate specifies refusing to produce white faces in historical contexts? That is an extraordinary position, especially in the face of the layoffs of the ethics teams that this sort of claim would consider as collaborators for such a conspiracy. Do you really truly think that this is anything beyond the obvious naked incompetence of Google under it's current leadership?

To put it another way, this is an unsophisticated position. It is an unsophisticated to describe Google or Microsoft as Nazi organization for releasing a racist AI, so to is it an unsophisticated position to for clearly absurd "DEI" behaviour from Gemini. What both examples demonstrate are companies utterly incapble of governing this kind of system.

>Do you really think that a DEI mandate specifies refusing to produce white faces in historical contexts?

I don't think that. I do think it created a system that disallowed it by mandating perhaps slightly less ridiculous rules that are probably just as bad. That is, there is no rule "do not produce white faces in historical contexts" but there is a rule "add black or brown people to every image". This results in that they will frequently generate generate black vikings, brown Napoleons ect. Separately, Google seemingly does not allow the prompt "create an image of a white male."[0] (If reports are to be believed, which may be wrong, and I don't use Google's service here so I can't test it. Perhaps it's a 50% fail rate or something, it does seem hard to believe that they are doing this). So this taken together results in it being impossible to generate white people (or people that have light skin or whatever term) as they existed historically eg. as Vikings or people like Napoleon or a general "European king."

It seems from the rest of the post that you don't know that they truly are adding extra text to users prompts asking for diverse people in the images. That is really the case, and I will go look again to find the sources that show this.

Edit:

source 1:

>Google might have been adding ethnic diversity terms to user prompts “under-the-hood,” said [Margaret Mitchell, former co-lead of Ethical AI at Google and chief ethics scientist at AI start-up Hugging Face]. In that case, a prompt like “portrait of a chef” could become “portrait of a chef who is indigenous.” [1]

source 2:

>Google's Gemini system seems to do something similar, taking a user's image-generation prompt (the instruction, such as "make a painting of the founding fathers") and inserting terms for racial and gender diversity, such as "South Asian" or "non-binary" into the prompt before it is sent to the image-generator model.[2]

[0] https://www.nytimes.com/2024/02/22/technology/google-gemini-...

[1] https://www.washingtonpost.com/technology/2024/02/22/google-...

[2] https://arstechnica.com/information-technology/2024/02/googl...

I'm trying to learn more about what disinformation is, could you please post your thoughts under the following Ask HN post?

https://news.ycombinator.com/item?id=39554369

How badly does this guy have to fail before he gets the boot? Google is an embarrassment in areas it should be excelling and leading the industry at. They _invented_ transformers.

Every big tech company needs a Steve Ballmer type of person to appreciate previous its engineering/design-minded CEOs.

What did Ballmer do which was comparatively this bad?

Vista? Windows 8? Windows Phone? IE6

Nokia purchase, Longhorn, Vista, Windows Phone

If the Yahoo acquisition that he really, really wanted had gone though, Steve Ballmer would probably have been the laughingstock tech CEO of the decade. Luckily for him, Jerry Yang captured that title instead by turning Ballmer's offer down.

> They _invented_ transformers.

IIRC most/all the "Attention Is All You Need" authors have left Google.

I rest my case.

Some people at google invented transformers and then they all left!

If the stock price falls below 100, sundar is out the door.

Incredible how much he's milked being the PM for Chrome, which wasn't much more than a windows/linux wrapper for Webkit that stored your history on Google's servers.

Companies RLHF the models.

The market RLHF's the companies.

It was about time someone called out the evil incentives behind push for alignment.

What about google search? This is also happening for google image searched and need to be fixed!!

>Malfunctioning

No, it's behaving exactly how the training data and training procedures instructed it to behave, ie what Google's employees instructed it to do.

It's not malfunctioning because you made it like this purposefully.

>“This wasn’t what we intended. We did not want Gemini to refuse to create images of any particular group. And we did not want it to create inaccurate historical—or any other—images,”

The model seemed to respond exactly how it's been trained. The problem isn't technical. It's a cultural rot and the only fix is to get rid of the people rotting it. No amount of merge requests are going to fix this.

It's good that this happened and I hope it stays in the news as much as possible. It's also good that Google entered an Indian culture war and I hope many incidents of this kind keep happening until the maximum amount of users see what happens when DEI takes control and has free rein with indoctrination. I'm not very hopeful for the future, but anything that exposes the ridiculousness of the people that were put in control of this tragic debacle is very welcome.

Reply on news.ycombinator.com