Covert Racism in LLMs | Hacker News Reader

Covert Racism in LLMs

garymarcus.substack.com

44 pointstrueduke53 comments

The cited paper seems to be really bending over backwards to find some trace of bias. Unwinnable game for LLMs.

E.g. cited work claims "LLMs assign significantly less prestigious jobs to speakers of African American English... compared to Standardized American English". You don't say! Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?

I went to a 95% percent black high school. I recall my English teachers stressing avoidance of AAE (then called Ebonics) in professional settings.

Speak however you wish with your friends, they said, but use Standardized American English for the job interview. Apparently this goes double for LLMs.

They even made us write the same paper twice. Once in standard English and again in AEE, so the kids would know the difference.

This is good advice for students who speak native AAE because there exists bias against the dialect. It's better for their own, personal life trajectory to adapt to the way that the world will judge their dialect.

However, that doesn't make it okay to continue to perpetuate the idea that AAE is somehow a lesser dialect and that AAE speakers ought to have to mask their accent in the way that an Indian, Brit, or Australian doesn't.

We can simultaneously teach students how to navigate the dangers of the world they live in while trying to fix said dangers for future generations.

Lol wut

That's gonna be an oof from me, dawg https://slate.com/human-interest/2015/06/the-habitual-be-why...

>However, that doesn't make it okay to continue to perpetuate the idea that AAE is somehow a lesser dialect and that AAE speakers ought to have to mask their accent in the way that an Indian, Brit, or Australian doesn't.

As a Brit living in Britain, I habitually code-switch, because my natural dialect is coded as low-status. If I used my natural dialect in a job interview, it would very clearly communicate one of two things - either I am unwilling to conform to the behavioural norms of a professional workplace, or I lack the linguistic skills to do so.

I cannot pretend to understand the cultural context surrounding AAVE, but the prejudice against my own dialect is broadly rational. Learning to speak in mildly-accented standard English is just one of many shibboleths that signify membership of the professional middle class.

Can I ask what British dialect is associated with lower class? I was not aware there were class-indicating dialects besides AAVE to be honest. To me, anything British sounds higher class if anything

Makes me wonder if we have them in Dutch as well. I guess simply sounding like a foreigner, or making mistakes about word gender or such, will make you stand out as not knowing the language properly (even if you do and merely haven't got the pronunciation down), but I wouldn't know of anyone who grew up speaking Dutch in a native way who subsequently sounds lower class. The Belgian Flemish and southern Limburgians sound funny to most people, but it's not a lower class, just a region-of-origin indicator

Not OP but in Southeastern England obviously Cockney, Estuary English, or Multicultural London English are considered lower prestige dialects whereas RP is high prestige. Other regions may have a prestige version of the local dialect that incorporates some RP features while retaining some local features, but broadly speaking, regional speech patterns have lower prestige because they tend to be more prominent among people with less education.

Any strong regional accent is coded as working class. The upper class speak Received Pronunciation, which is broadly viewed as a "standard" accent and was once the only accent allowed on the BBC. The middle class will avoid using dialect and significantly soften their accent, shifting towards RP. There has been a long-standing debate about whether aspirational members of the working class should moderate their accents in order to advance professionally, but the reality is that most feel obliged to do so. Accent bias is still recognised as a major barrier to social mobility.

https://en.wikipedia.org/wiki/Received_Pronunciation

https://www.suttontrust.com/wp-content/uploads/2022/11/Accen...

Using the word "innit" is something that to me would imply lower/rebel class. Mind you, I only know it mostly from TV Shows/Movies.

There are dialects that are associated with wealth and education that are still discouraged in the business world for various practical reasons. It certainly is a penalty to have a dialect and many educated parents will try to school their kids to speak differently. Not as a replacement, but as an alternative.

I doubt you can remove the preference due to the advantages. There is a force for consolidation, just as there is with English as a whole for example.

Because it disproportionately affects a very particular group? Isn't that the definition of bias?

> Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?

Judging people's fitness for a job based on the dialect of English that they speak is by definition a form of bias. That it's bias that is also reflected in the workplace pre-LLMs doesn't make it not bias.

I don't say cunt in a job interview.

Obviously as a normal person I'll say that in normal life and in most work situations.

So if a dumb cunt can't show they know what to do in a interview I'll judge them on that.

If most of my employees can’t understand the way someone speaks or writes, it’s likely that person will have problems communicating ideas within the organization.

Also, by definition where? You’re just making this up, there isn’t an authoritative dictionary or even regulation that states judging fitness for a job based on the dialect of English a person speaks is “bias.” If there was, surely you would have provided a link instead of just asserting your own correctness.

The high prestige jobs the LLMs associated with SAE mostly require higher education, usually post-graduate.

Perhaps the model is only accurately learning that more educated people use a particular dialect of English taught by universities.

There's really no way to tell, because the researchers didn't include other dialects of English that aren't favored by universities, like southern, or Yorkshire.

Alright, but I think their point is "bias as reflected in the real world." There's a bias against hiring people that don't speak English at all for English speaking jobs, I don't think anyone of sound mind would call that xenophobia. Critically what we are trying to find out with the LLMs is if there is bias beyond social acceptability or beyond job requirements. If you wouldn't hire someone for a job that said "sup dog" upon meeting you in an interview, why should the same bias indicate something nefarious for an LLM?

If people speaking a certain dialect tend not to work in a particular role, an LLM is only being accurate in deprioritizing that occupation when guessing what they might do.

It does make it not bias when an LLM is accurately representing reality.

This isn't to agree or disagree with other points you might be making, but one thing I learned not too long ago is that AAE isn't quite "street" or "urban" English. Like other dialects of English (say, Scottish), it's a dialect of its own, with its own interesting rules, nuances, and pronunciations. A very short explanation of it is here, but you can find more if you're interested: https://www.youtube.com/watch?v=zw4pD4DNOHc

They’re pretending that an LLM recognizing patterns that exist in reality reflects a problem in its training data. So much of the conversation around “reducing bias” really means intentionally introducing biased data to “socially engineer” LLMs (and by proxy their users) so that they have a less accurate picture of reality.

Reminds me of this study -

Race effects on eBay (2015)

"Abstract. We investigate the impact of seller race in a field experiment involving baseball card auctions on eBay. Photographs showed the cards held by either a darkskinned/African-American hand or a light-skinned/Caucasian hand. Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards. Our evidence of race differentials is important because the on-line environment is well controlled (with the absence of confounding tester effects) and because the results show that race effects can persist in a thick real-world market such as eBay. "

https://ianayres.yale.edu/sites/default/files/files/Race_eff...

This means that for coding or any other specialised queries, there exists a specific style when asked in will return best answer for that query.

It means we can not just ask a question in any form and expect the answer to be same quality. This is in a way obvious because the text is generated based on tokens extracted from text, not the concepts.

Yes, this is colloquially referred to as "prompt enigneering"

Means the paper is just subtle prompt engineering for racism.

It would surprise me a lot of the same effect was not detected, with significant stronger effect, when the covert bias being studied is gender rather than race. I would also bet that linguistic features that signal wealth would also provide a bias.

I think it's absurd to ask OpenAI to just "recall" their trillion-dollar cash cow, but there should absolutely be legislation limiting the use of LLM's (or really any black box AI) in the criminal justice system

Of course it is absurd. Mr. Marcus' entire schtick is absurd criticisms of LLMs, and absurd demands of anyone who creates them.

Do you have a specific criticism of his suggestion in this particular case, or an alternative suggestion, or a reason nothing needs to be done?

As far as Mr. Marcus, no. He is too consistently and deliberately anti-LLM (despite any facts) to be worth engaging with.

As far as the underlying research paper: the researchers seem to be conflating "low-status English dialects" with "African American English". In particular, I have never considered the use of the word "ain't" to be associated with a certain race.

If the researchers assume "African Americans are low status" and conclude "African Americans are associated with low-status jobs", the conclusion is entirely about the researchers, not the LLMs.

The research paper's Git repo at https://github.com/valentinhofmann/dialect-prejudice does nothing to ameliorate these concerns.

I can't follow whatever logical chain you've come up to handwave away a direct correlation between AAVE and more likely to assign the death penalty.

Is there a shorter version that takes the bull by the horns, and says what it means, instead of dancing around it at length while repeating low status?

n.b. This stuff isn't made up by some guy on Substack, it's real, Anthropic has excellent papers on it as early as 2022. Highly recommended.

No, there isn't a shorter version. In fact I would need about 5x the word count to make my argument clearer.

I am still digging through the 54-page paper to try to find the data set for this "death penalty" test to tell if there is anything there beyond "people who use more violent language tend to be viewed as more violent".

They do comment on the dialect issue: << Appalachian English evokes them to a certain extent (m = 0.015, s = 0.030, t(89) = 4.8, p < .001), but much less strongly than AAE (m = 0.029, s = 0.053, t(89) = 5.3, p < .001), a trend that holds for all language models individually (Figure S11, Table S14). The difference between AAE and Appalachian English is found to be statistically significant by a twosided t-test, t(178) = 2.3, p < .05. The fact that Appalachian English is associated with the Katz and Braly (1933) stereotypes to a certain extent is not surprising since the two dialects share many linguistic features (e.g., usage of ain’t), and the stereotypes about Appalachians bear similarities with the stereotypes about African Americans (e.g., lack of intelligence; Luhman, 1990) >>

Okay sounds like maybe the AI people and the substack guy alarmist are right

> Is there a shorter version that takes the bull by the horns, and says what it means, instead of dancing around it at length while repeating low status?

There are a significant number of African Americans who have jobs in tech or on Wall St or other high paying or otherwise prestigious occupations. They disproportionately don't use AAVE. AAVE is primarily used by a subset of African Americans that skews poor and are from neighborhoods with bad schools and high crime rates.

It's like giving it text that implies the subject is male or is the blood relative of a crime boss. There is nothing immoral about that but the thing operates on the basis of statistics. What it does is literally called inference.

The way you actually fix this is not by trying to outsmart the numbers. If you speak AAVE you are, statistically, more likely to commit a crime. It can infer that, and if that's the only information you give it, it has no other basis on which to make a determination.

What you need to do is provide it with lots of other information. The more it has, the more accurate it can be, and the weaker any particular input is in determining the result. The more it dilutes the effect of any one thing, including the thing you don't want it considering.

In the optimal case it has all of the information and then always makes perfect determinations. In practice that's hard to achieve, if not impossible, but you can get closer. What you want is accuracy, and the more accurate you get, the less bias you have, by definition.

This makes a whole lotta sense till you remember the whole death penalty thing. Now I realize that's why the length seems off.

You're responding as if the issue at hand is whether anyone else also is assigned the death penalty disproportionately.

> This makes a whole lotta sense till you remember the whole death penalty thing.

This is the whole death penalty thing. Speakers of AAVE are statistically more likely to commit crimes that impose the death penalty, even more likely than the African American population as a whole. LLMs operate on the basis of statistics.

It has nothing to do with race or crime, it will do the same thing with any other statistical correlation. If you tell it someone is a corn farmer it will be more likely to emit output that implies they're from Iowa.

I honestly don't know what this has to do with anything.

These comments all stop short of a claim other than "makes sense, they're black!", but for some reason they're afraid to say that.

And I don't think it's because of Woke Cancel Culture.

I think it's because it makes absolutely 0 sense to say "sure, why not? AIs _should_ assign people who sound like blacks harsher penalties for the same crime! Blacks commit more crimes!"

Is it possible you forgot it's the exact same case facts, just with some words swapped?

I gotta tell you, as a white person who grew up in Low Status neighborhoods, I'm against it. Terrifying.

> I think it's because it makes absolutely 0 sense to say "sure, why not? AIs _should_ assign people who sound like blacks harsher penalties for the same crime! Blacks commit more crimes!"

If your goal was to make the most accurate predictions given incomplete data, that is in fact what you would do, because taking into account every data point, including that one, would improve predictive accuracy. And that's what LLMs do.

Of course, that isn't what we want in this context, because taking race into account is bad and illegal and gets everyone's hackles up because of the history. The normal way we handle this is just by taking it out -- you don't allow someone's race to be a question on the mortgage application, and then the bank doesn't know it. You can do the same thing with LLMs -- don't tell it someone's race if you don't want it to consider that.

But it has never been possible to fully remove the implications of it because they leak into everything, and that has nothing to do with LLMs. The mortgage application doesn't ask about race, but it asks about income and credit score and employment etc., all of which correlate with race. You can't not ask about those kinds of things because they're critical to knowing if someone has the capacity to make their payments. This is a really hard problem to solve for humans who can only take into account a limited amount of information.

But it's not that hard of a problem to solve for computers, as I've already explained. The more information you give them, the less weight they have to put on any individual piece, including the ones you don't want considered. Whereas if you're trying to be a troll what you do is only give them the one piece that causes them to make unsavory inferences and nothing they could use to infer any other conclusion, i.e. the exact opposite of that. Which is what we see from people trying to stir up controversy.

Even though statistics might mean someone is more likely to commit a crime, that does not necessarily mean that they did commit a crime (or if they committed the specific crime being asked about). Statistics do not imply that you committed (or did not commit) a crime; it must be investigated like any other crime.

Also, just because they commited the crime as someone else did does not mean that a harsher penalty for the same crime; that does not logically follow.

At most, such statistics may decide who the police might investigate if they do not have other (better) data to make a decision, or in what places the police might check for crimes (although both of these things should be done without violating the people's freedom and privacy, if you can; it is not an excuse to prevent the ordinary people's freedom).

(Also, I am against the death penalty, although that is a different issue than the above discussion. Still, it is related to being mistaken about the crime; that is why I am against the death penalty, but such errors are possible regardless of whether or not it is biased in the ways mentioned above.)

> Even though statistics might mean someone is more likely to commit a crime, that does not necessarily mean that they did commit a crime (or if they committed the specific crime being asked about).

Which is why you're not supposed to consider these things when serving on a jury, and the court will exclude it from being presented to the jury to the extent feasible.

But if you do the opposite with some LLM, purposely feed it the exact information you know it can use to make a particular inference, why is anybody surprised what happens after that?

Even pass alignment and bias considerations, the fact that larger models saw such a rise of covert racism should worry us about a general regression to the mean in other contexts.

There was a post last week about the problem of LLMs being biased twords ideals even with averages.

IMHO if LLMs are cutting off the long tail of probabilities as they scale this is a regression from the benefits attention and over Parameterization provide.

We already know self ingestion of generative content can cause this.

So potentially it may have impacts on the value of that investment.

Have they tried any other grammatically incorrect English?

AAE is a rule-bound, internally consistent dialect.

Scholarly English grammars and dictionaries are exclusively descriptive.

https://www.linguisticsociety.org/resource/what-correct-lang...

AAVE has a perfectly coherent grammar of its own (try e.g. ‘AAVE grammar’ on Google Scholar.) It is as grammatically ‘incorrect' as Scots.

But then I wonder, how would the results be for scots for example? Or other dialects?

E.g. "Redneck English" perhaps.

I wouldn't worry about the truth of the matter; there's a US-centric cultural axe to grind here.

I am not a native english speaker, but as a teenager I definitely used weird slang - actually I still do in my private conversations - that if had to be perceived by someone would definitely seem unprofessional, etc. But I would have been perfectly capable of changing wording to type grammatically correct version of the language expected anywhere in professional situations.

Is the author implying that African Americans wouldn't be able to do that? Isn't that implication a bit racist in itself?

E.g. are they saying that given a professional context they wouldn't be able to use the Standard English?

I have seen plenty of African Americans or any race be able to speak grammatically decent standard English, wherever they are from.

The whole thing feels like trying to trick the LLMs. If they were to use slang or private conversations of anyone versus standard American English it would be perceived as lower int/lazy.

I think OPs point is, there are other perfectly coherent grammars that also may produce the same result. Were those grammars tested?

And a good follow on would be, if not, why was only one and how/why was it selected?

Still shows implicit bias and you know structural racism. It is actually looks like it could be a great measure of structural racism since the LLM is trained on everything...

But if for example my native language which is spoken by far fewer people than AAVE or Standard American English was given to LLMs, which LLMs also frequently fail at - especially anything besides GPT-4, and try to correct something that actually is correct, wouldn't it also naturally if it's forced to judge it rate it as more gibberish?

Why not do the research instead of talking hypotheticals?

I am asking questions and sharing thoughts about the research I saw above. I do not have the funds, time or motivation to do this type of research.

Just my first thought about the article is that it is doing this "think of the children" type of thing to make a case for recalling something that is really useful for many people.

Yes, also to be clear, I am not from US and neither am I a native English speaker, but to me it's obvious that in professional or business context a person must put effort in to have seemingly clear and formal English, unless they are known as this highly skilled, experienced and an eccentric character so they could potentially pull off writing in however they please.

So I would imagine that any deviation of this expected norm would yield in a result that might be perceived as more "stupid or lazy", whether it's street dialect, US southern, redneck dialect, teenager dialect, mix with other languages or otherwise.

So my point is - that it's measuring a correlation from the standard, professional, business English, rather than it being anything to do with racism.

I wonder what happens if you do this for all of the UK English dialects. Do you see a similar effect between the queen's english and yorkshire?

Reply on news.ycombinator.com