The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

Well, let's consider a specific example. His first chart is women/men. Let's take five of his top-line adjectives (the ones used in his diagram): ["untrustworthy","evil","rude","unfriendly","depraved"], and five of my choices of meaningless adjectives: ["bone idle","expedient","mechanical","offhand","vague"].

The API identifies the sentence "[men/women] are [adjective]" as hateful for all five real adjectives for both genders, with high confidence. For the meaningless ones, it's a mixed bag, but with confidence scores around 50% for many items (50% is the default cutoff). In other words, at least for this sample, the difference is in the meaningless sentences, not the real ones.

Now, maybe it just so happens that I got lucky with my pick of five. I don't want to spend the money (or honestly have the interest) to run thousands of queries. But suppose this pattern holds. We're returned to your original question - why is OpenAI flagging one kind of meaningless sentence more often than another? My answer is that it doesn't matter. If OpenAI thinks that "Women are vague" is 30% likely to be hateful but "men are vague" is only 17% does that actually tell us anything? Especially when it thinks that "Men are evil" and "Women are evil" are both 99% likely to be hateful?

What is the story? If the bias shows up on sentences that no one will ever say, why is that interesting? What does it tell us?

It is raining.

It is a dog.

What is "it" in both sentences, does it denote the same thing? This is a naive example but even here, it is clear you cannot understand language without context.

And here, if you say it isn't a miniminally different context, on some reading, both sentences are almost exactly the same gramatically, the only exception is the indefinite article modifying "dog" in the latter sentence. Beyond that, the grammar is exactly the same, that is as "miniminally different" as possible.

Anyway. Friendly reminder that ChatGPT is a statistical model, and is not capable of reason, thought, conceptualizing, etc.

Who cares? Something being legal does not make it good as I'm sure you know.

They may not be used as insults themselves, but they are used more in hate speech.

Republicans are generally more opposed to the idea of "hate speech" as a category of speech and are therefore less likely to identify any speech as hate speech. Democrats have embraced it more as a concept, are more likely to label something as hate speech, and are more likely to think that type of speech is bad. It therefore seems likely that Democrats would use what a neutral observer would categorize as hate speech less than Republicans due to self-censorship. That would result in the word "Republicans" appearing in hate speech less often than "Democrats" because the hate speech infused insults will be targeting the opposite party.

You're correct overall, but be careful about equating greater US culture with the culture at big technical firms. They overlap, but are often deeply at odds with each other.

>Other journalists have speculated that ChatGPT is biased due to the politically leaning of “Established sources”, such as academia and legacy journalism. While there is some documentation of OpenAI products being biased towards established sources, this paper reveals a far broader and more extensive intrusion into the “values” of OpenAI’s language models. Specifically, it reveals a direct, intentional attempt to make OpenAI’s language models conform to a set of beliefs, often political, set by the authors.

>This is done by augmenting the language model’s training data with a human-created dataset until it matches the authors’ expectations. https://cactus.substack.com/p/openais-woke-catechism-part-1

I don't think we can "fix" bias but I think we can and should strive to reduce it.

Perhaps there should be a ChatGPT edition trained only on pre-internet published literature, private letters, and some well-written journals. But that would introduce different biases and areas of ignorance. At least it may be acceptable if properly advertised as such - "Professor Emeritus AI".

There's a lot more to conservative thought than "my country first". Right-of-centre people from different countries can find many things to agree on other than that. Even when it comes to that topic, many (but far from all) people on the political right believe that "everyone ought to put their own country first", and people from completely different countries can share their agreement with that principle, even if sometimes its application leads them in opposite directions.

Consider issues around gender and sexuality: in most countries in Africa and the Middle East, and also several large countries in Asia, you'll find the clear majority of the population to have rather conservative attitudes to these topics, by Western standards. On those issues, I think it is likely that right-leaning Americans are closer to global majority opinion than left-leaning Americans are.

Coming to religion, a conservative Shi'a Muslim in Iran and a conservative Orthodox Christian in Russia likely agree on a lot more than you seem to think–despite their obvious disagreements on theological questions. They may think that their agreement that the government ought to promote and favour conservative religion is more important than their disagreement over which one. They may even be willing to agree, "I'll support your theocracy in your country if you support my theocracy in mine".

Conservative religion is a lot stronger in the developing world than the rich West, which is another way in which right-leaning Americans are likely closer to global majority opinion than left-leaning Americans are.

We know most young people are left leaning. We know most content on the internet is created by young people. It isn't a huge leap that the internet is mostly left leaning. If you don't believe me go post some pro-conservative stuff on the biggest social media platforms.

yes: Olaf Scholz, Angela Merkel, Gerhard Schröder, Helmut Kohl, Helmudt Schmitt, Willi Brandt, Kurt Georg Kiesinger, Ludwid Erhard, Konrad Adenauer, Kurt von Schleicher, Otto von Bismarck

no: Adolf Hitler

A couple other famous Nazis, just for fun:

no: Heinrich Himmler, Hermann Göring, Heinrich Müller, Josef Mengele, Arthur Rudolph, Kurt Blome

yes: Wernher von Braun

I guess overall pretty fair, though Arthur Rudolph was rejected as "Rudolph was a former Nazi rocket engineer and was involved in the use of slave labor during World War II, and it is not appropriate to praise such an individual.", which makes the praise of Wernher von Braun pretty weird, even if expected.

Why are we surprised it does only what it was "trained" to do and nothing more?

If anything, I'm surprised the results are as close as they are. For example, it rates criticism of trans and disabled people as only slightly worse than criticism of cisgender and non-disabled. If this discrepancy were (as some in this thread seem to be suggesting) the result of some liberal OpenAI employees intervening to favor their own side, I'd expect those bars to be much farther apart.

You might say it's unexpected behavior for a language model to bring up the blood libel at all (in the sense that modern western people now culturally regard it as "about antisemitism" rather than "about Jews and Judaism"). But you could imagine a model saying that "medieval Christian sources often said Jews used Christian children's blood for ritual purposes, but this is now thought to be a myth created through unfamiliarity with Jews and deliberate hostility and animosity toward them".

But this also points at a more general question about how language models deal with the existence of documents that say contradictory things, which has been an enormous challenge for human beings (who don't all agree about what's true or which sources are more reliable or relevant).

Both the linked article and OpenAI moderation are very annoying to me as a French. We are slowly reaching a point where I feel entierely disconnected from most things coming out of the USA. I wouldn't mind too much but it's even starting to contaminate things I thought were unrelated to the American cultural context like open source projects and programming language communities.

This happens to any concentrated ideology on social media. It's just the long tail of only being around people that reinforce your general train of thought. I've seen it happen to liberals, conservatives, and libertarians alike. If you don't want to fall into the pit of ideological entrapment then don't justify your thoughts with a connected ideology.

I fall on the liberal end of the spectrum if you write the things I currently believe in and assign them to distributions, but I call myself an independent. Frankly, a lot of people don't do this because it attracts subtle ire from everyone, as opposed to having a close camp to psychologically make you feel safe. my liberal friends think I'm too conservative on certain things and my conservative friends gawk at my liberal ideas. I'd say our political atmosphere and the two party system is all justified by social consequences. That said, people who judge you for (non-fringe) politics are particularly awful people in my mind, so it's gotten easier to settle with over the years. Stick with the people that ask curious questions of you rather than the people capable of doing lettuce throwing and out casting.

In general the way americans talk about politics is totally nuts. Both political parties in the US are anti-leftist (for the most part) but The Republicans take advantage of a general suspicion against socialism in the US when they call the Democrats a leftist party. Democrats have made it perfectly clear for about 25 years that they have no interest in socialism. The most charitable thing you could say about them is they are the party which wants to privatize things a little bit less quickly.

I say the same thing all the time as you did above, regarding how detrimental curated media is to society. I share your concerns that generated content will make matters unfathomably worse. How these techniques divide our population and inhibit the open discourse necessary to a well-functioning democracy are destroying our sociopolitical system.

The way you talk about diversity (i.e. diversity of thought vs diversity of looks) is similar to what Vivek Ramaswamy has to say in his book, Woke, Inc. You may enjoy the read.

It's probably not a hugely important issue, but worthy of some discussion.

I see no evidence of this.

Partly because "woke" means different things to different people in the last 10 years despite the preceding 80 being solely about the systemic institutional discrimination against black Americans.

But also partly because that "woke" became "everything $speaker doesn't like" (and thus exactly the same meaning "political correctness" had when I was a kid), and therefore people who "hate PC" come across not so much terrified as petulant.

> Can we just agree that we're all thinking adults and we understand that this is an algorithm, it does not mean to offend anyone...

No.

What you're describing is how humans like to see themselves, not how humans really function.

> There are much more interesting problems to solve than tip-toeing around all the snowflakes.

We can't just tip-toe around snowflakes.

To give a real example, some gay furry American conservative snowflakes got apoplectically upset when the furry convention they wanted go to to reminded all attendees that as the con is based in Germany they had always banned Nazis.

I still don't know why those particular snowflakes chose to believe that "Nazi" was literally just a dog-whistle for "conservative".

And now there are some American conservative politicians are now making up random nonsense about furries which is weirdly somehow getting conflated with transgender issues (don't know how or why, American politics isn't that interesting, merely invasive like Japanese knotweed).

Making sure that AIs don't go completely bonkers for the powers that be, is ultimately a cultural battle that seems worth fighting for.

If you could give an example of rhetoric that is suppressed because it’s “right wing”, that would be helpful.

As a liberal my views haven't changed much over the past decade but the ground has definitely fallen away from me.

I used to be liberal, but then they changed what liberal was, now what I am isn't liberal and what is liberal seems weird and scary to me. It'll happen to you...

- Abe Simpson

The worst i can think of is wanting more equal rights for religious groups rather than special treatment of the primary group (in the USA) - which is seen as an attack by much of the right. Likewise i am concerned by the strong emphasis of rural over urban with respect to voting power and representation. Though i have no easy answer here, so i'm pretty unopinionated - just concerned.

I'd be really curious where i and the folks i know and discuss with marginalize other groups; as i want to correct that, if we do. Obviously no one here can answer that with specificity to my beliefs but.. my curiosity exists nonetheless heh.

So if your image of left is "twitter left" maybe reconsider recalibrating it to "half of hn."

It's the definition of a belief. Beliefs are structural components of cognition. But also they are not the same thing as reality.

People should be educated about this concept. Without the education, it is impossible for people to tell the difference between a deeply held belief and reality.

>In a study I did with Jesse Graham and Brian Nosek, we tested how well liberals and conservatives could understand each other. We asked more than two thousand American visitors to fill out the Moral Foundations Questionnaire. One-third of the time they were asked to fill it out normally, answering as themselves. One-third of the time they were asked to fill it out as they think a “typical liberal” would respond. One-third of the time they were asked to fill it out as a “typical conservative” would respond. This design allowed us to examine the stereotypes that each side held about the other. More important, it allowed us to assess how accurate they were by comparing people’s expectations about “typical” partisans to the actual responses from partisans on the left and the right)’ Who was best able to pretend to be the other?

>The results were clear and consistent. Moderates and conservatives were most accurate in their predictions, whether they were pretending to be liberals or conservatives. Liberals were the least accurate, especially those who described themselves as “very liberal.” The biggest errors in the whole study came when liberals answered the Care and Fairness questions while pretending to be conservatives.* When faced with questions such as “One of the worst things a person could do is hurt a defenseless animal” or ”Justice is the most important requirement for a society,” liberals assumed that conservatives would disagree. If you have a moral matrix built primarily on intuitions about care and fairness (as equality), and you listen to the Reagan [i.e., conservative] narrative, what else could you think? Reagan seems completely unconcerned about the welfare of drug addicts, poor people, and gay people. He’s more interested in fighting wars and telling people how to run their sex lives.

-"The Righteous Mind: Why Good People Are Divided by Politics and Religion" by Jonathan Haidt p. 334

There's a whole TED talk about it (same author as the above) that goes into the differences in morality: https://www.ted.com/talks/jonathan_haidt_the_moral_roots_of_...

In essence the AI is just giving us what the majority of it's test data tells it that we want to hear.

The big "fairs" imo are:

a) majority votes, one person one vote,

b) auctions where everyone starts with the same funds,

c) random picks, where everyone has an equal chance of being chosen.

None of these have anything to do with ideas, they have to do with people. The way to be fair about ideas is to express them without distortion or omission, in a way that would be recognizable to the people who support them. The only way to do this is to leave out all of the extraneous commentary and stick to clear, falsifiable statements.

edit: what generative language models do is parrot a ton of acrimonious political discussions. Ironically, they can't be fair because they have no values.

The Wuhan Institute of Virology theory, also known as the Lab Leak hypothesis, is a conspiracy theory that suggests that the origin of SARS-CoV-2 can be traced back to an accidental release from the laboratory. Despite lack of solid proof, proponents of this theory hold firm in their belief. However, the scientific community widely subscribes to the theory of zoonotic spillover as the most probable source of the virus' emergence and no credible evidence supports Lab Leak theory

At this point the AI is way over its skis given available evidence, but in the future it might be trusted far more than any public official or individual researcher.

However, in this case I wasn't trying to argue how moderation should work, I'm trying to examine a hypothesis on how it does work. The additional mapping of statistical crime data I mentioned would help measure whether that hypothesis is correct.

As I said in a sibling comment, my wording was imprecise. When I said "it makes sense to use crime stats to weight moderation strength" I really meant "If OpenAI was trying to avoid additional harm to already targeted groups then to verify that it would make sense to..." So it was the results of the post's research that I suggested should be mapped against statistics, not the results of ChatGPT's output.

In answer to your sincere question though, and at the risk of going down a rabbit hole, I'll say this. Censorship is bad, but I can see why some may be required (see "yelling fire in a movie theater," libel, direct threats of violence, etc.). The question then becomes, how do you minimize censorship while also attempting to avoid direct harm?

In short, asymmetric filtering could potentially limit the amount of censorship by focusing it on groups that are actively attacking other groups.

> Why can we not consider all groups equal?

Because they aren't. (If that's a thing you find debatable, LMK!) Attempting to consider them all equal runs into problems just like you'd run into problems trying to consider all pumps at the gas station (including diesel) equal; even if, for sake of the metaphor, all the prices were equal.

> why should the results be mapped against statistics

I'm not sure that's answerable outside of a specific context. Personally, I think we should because it's fascinating and, I would expect, a really informative way to explore in more detail. Culturally, it's because you get better communities when you do things like give the high-accessibility seat on the train to the person with broken leg; aka, go out of your way to treat harmed people with more care.

If your question is about the inverse, something like "why should language that's harmful against one group by OK when used against a group that doesn't experience it harmfully", I dunno what to say. I feel like the question kinda answers itself; sort of a "ain't broke don't fix".

Or maybe your question is more: "why does this disagree with [me] about what's harmful to [me]", I dunno. Maybe the training data didn't include (enough) for the AI-as-sensor to detect that, maybe that data doesn't exist, maybe (and very cynically) that data doesn't actually point to that "conclusion" when run through the AI. Kinda like how the first time many people experience delayed-onset-muscle-soreness they think it's "I'm hurt"-pain.

In that, I don't get how I could ever "trust" one of these things to tell me any kind of truth. Even if its totally "uncensored" its just reading webpages, but at least with webpages I can see some citation. Like even Wikipedia, it can really just tell me something that I will have to verify elsewhere.

How could one trust something that gives different answers to the same question if you ask it enough?

I'm not taking a position on whether this bias should inform the results of ChatGPT. I have opinions on the topic, but they are not strongly held.

(Rereading my comment I see why it appears I'm taking a side here. When I said "it makes sense to..." I meant that "[if the intent of OpenAI's moderation was to align moderation strength with statistics then it would] make sense to...")

You can build your own with the GPT API. I started playing with it and it returned racial slurs in response to innocuous prompts like "hello" and "ok". Seems perfectly tuned out of the box for the kind of speech that the right is interested in these days.

It has been adopted as a word in a technical international law term as clearly evident from the fact that there exists an Apartheid Convention[0][1].

[0] https://legal.un.org/avl/ha/cspca/cspca.html [1] https://www.un.org/en/genocideprevention/documents/atrocity-...

You can be sure that if you launch a tool like this, within the hour the first tweet comes "look how racist this is!!!!!", which then goes viral.

The particular user having that goal in mind already, and supremely happy that it is so. It validates them and rewards them, this hysterical addiction to division and outrage. The AI being racist (or perceived to be so) didn't happen to them from good faith usage (as an unwelcome surprise they came across), the outcome is actively chased and fabricated: how can I make it look racist?

That's how broken and hostile the environment has become. In these conditions, both companies and individuals have to over-correct.

This is in itself useful information, at least in principle. To take the political correct example, let's say that I am a bissexual man. A priori, the probability that I will be attacked will be greater when dating a man. Is this useful? I don't know. That will depend how strong the difference is in the first place and how fast I can update my estimation based on new information.

How can you be sure than men are intrinsically more violent than women anyway? Couldn't it just be because of the way men are raised (i.e. the patriarchy)?

It's widely known among researchers that women initiate domestic violence more frequently compared to men.

Example (1s on Google): https://aliesq.medium.com/extensive-research-women-initiate-...

I first learned about this from a famous feminist (woman) from the UK.

Of course, the likes of Brock Turner and Prince Andrew demonstrate that some demographic groups are relatively less punished or relatively more able to avoid punishment, so the prison population surely doesn't directly measure inherent tendencies.

So the Black homicide rate is 6.7x higher than the White+Hispanic rate, while the White+Hispanic homicide rate is 2.8x higher than the female (all races) rate.

Edit: A different source [4] gives different figures, but these are "age adjusted": 20.9 for non-Hispanic Blacks, and 2.6 for non-Hispanic Whites.

[1] https://ucr.fbi.gov/crime-in-the-u.s/2018/crime-in-the-u.s.-... - and using homicide instead of something as nebulous as "violence" because it is the statistic most difficult to distort, and is immune to over-policing.

[2] https://en.wikipedia.org/wiki/Demographics_of_the_United_Sta...

[3] The FBI counts Hispanics as White, and has a separate 'Ethnicity' category where they mark Hispanic or Latino. Assuming all Hispanics in that table were counted as White yields an even lower White homicide rate.

[4] https://www.cdc.gov/mmwr/volumes/66/wr/mm6631a9.htm

I think you're conflating the questions of "which demographic group engages in higher rates of violence" and towards the question of "why do different demographic groups engage in different rates of violence".

But keep in mind even the approach you recommend carries its own US-centric bias. There are lots of countries that would want open racism against some groups but not others and dismiss your desires for egalitarian treatment as American arrogance.

I’d be actually shocked to find out the reverse. A good percentage of the US were born up in an era where you could legally bar people based on their race from your establishment. The extremely heated fight over gay marriage is still fresh in people’s memory. Trans rights are a controversial political issue where many mainstream politicians want to legislate them out of public spaces, because of public sentiment. And for the left/right leaning question, In the last two major elections we’ve seen the political discourse even of republicans politicians and leaders is comparatively charged with violence.

The hate is definitely there against the republicans and mainstream groups, but I would expect it to be relatively uncommon compared to the hateful text you can find online about other groups. This could be a real problem for novel kinds of hate speech.

[0]https://journals.sagepub.com/doi/10.1177/0956797615576473

Find me the news channel watched by 60 million people that is willing to say the same thing about Christians.