Hacker News Reader

Comment by crackercrews | Hacker News Reader

Good question. For a start, don't pretend that Nazi soldiers were a multiracial bunch. And don't do whatever Google did to generate clearly-incorrect output like this.

View on HN

Sure. That's a major over-correction. But there are existing and known biases in data sources that need to be accounted for - you don't want to further perpetuate those.

I think it's obvious that Google went to a ridiculous extreme in the other direction, but there does need to be some amount of work done here. For example, we repeatedly have seen that just changing the name on a resume to something more European sounding can have significant impact on callback rates when applying to a job, and if you trained a model to screen resumes based on your own resume result data, this bias could be picked up by the model. That's the sort of situation these are meant to correct for.

That multiracial Nazi soldiers thing wasn't baked into the model: it was a prompt engineering mistake, part of the instructions that a product team were feeding into the Gemini consumer product to tell it how to interact with the image generation tool.

Here's a similar example from the DALL-E system prompt: https://simonwillison.net/2023/Oct/26/add-a-walrus/#diversif...

"mistake"

You keep using that word. I don't think it means what you think it means.

But seriously; a "mistake" is usually something that cannot be foreseen by a group of people reasonably talented in the state of the art.

This product release was so far from a "mistake", that it isn't funny. It was spectacularly well tested, found to be operating within design parameters, and was released to great fanfare.

They expressed delight in their product, and actually seemed surprised that there was a backlash by the great benighted unwashed masses of their lessers, who clearly couldn't be expected to understand the elevated insights being produced by their creation!

So: not a "mistake". Institutional Bias, baked into a model. Remember: a system's purpose is what is does, not what you think it is supposed to do.

It wasn't baked into the model. It was in the prompt. The model and the product built around it are not the same thing.

If end users can't access the model without going through the prompt system then that distinction doesn't matter.

From an end user point of view I agree.

As someone who works either these models as an engineer, I think it's important to understand that a feature implemented as part of the user-facing UI to a model is irrelevant to the work I do with that model via an API.

I don't see how you can solve that problem without inserting bias in a different direction.

There's a difference between inserting bias and allowing a real-world pattern to exist in AI. There may be reasons to dislike these real-world patterns, but that doesn't mean that allowing them to exist in AI is inserting a bias.

For example, if you ask AI to write a realistic story about an NBA team, and it comes back with a team with stereotypically Asian named players, that would be unrealistic. If it came back with a team with stereotypically Black named players, that would be fine. Does it reflect a real-world pattern? Yes. But not changing the algorithm to generate diverse names isn't inserting bias. It's letting AI reflect the real world, as it exists.

A basketball player named Yao Ming is unrealistic?

I used the plural. Have there ever been any NBA teams with more than one Asian on them? Certainly not starters.

I think this just pushes the unsolved part to the middle. We'll never have an undisputed definition of the real world, as it is.

Clear cases like chinese NBA players aren't contested, but ugly social issues with layers of abstraction and contradiction.

Reply on news.ycombinator.com