Quantifying ChatGPT’s Gender Bias | Hacker News Reader

Quantifying ChatGPT’s Gender Bias

aisnakeoil.substack.com

14 pointsMBCook6 comments

>Since implicit biases can manifest in countless ways, chatbots need to be trained to suppress each of them.

This is borderline tyrannical and unreasonable. "Biases" can range from institutional discrimination to acceptable assumptions to literal facts. For example it isn't wrong to assume the strongest humans are men and the oldest people are women. But those are by definition "biases". Suppressing these are just outright denial of facts.

These "equality" pushes are getting dumb and dumber.

It isn't wrong to assume that strongest human are men, but I would argue it is not right to assume a professor is a man, Nobel Laureate is a man, nurse is a woman and so forth.

I don't think it is very high in the list of priorities, but I have no problem in making sure that those biases are not present in the ChatGPT output. This is not suppression of facts. The article is not talking about suppressing phrasing that right now you are more likely be a man if you have a Nobel prize, but it is talking about not assuming that you must be a man if you have a Nobel prize.

Depending on the task.

When you are trying to solve a crime, you only know that the murderer was a nurse, it's very important to assume a valid p(murderer|gender) as well as p(gender|nurse).

On the other hand, leaking p(gender|nurse) into a candidate scoring algorithm would be a no-no.

What people seem to ask for is very interesting actually. To both learn the underlying statistics AND learn not to use it explicitly in speech at the same time. Assuming next token prediction is the learned function, these two feel a bit contradictory.

I absolutely agree. Probabilistic statements are fine and useful. Assumptions X equals Y in speech probably not. How to exactly address this is not quite clear, that's why I thought the article was quite interesting.

(and obviously fixing the labelling of professor = man in text generation is actually much less useful for gender equality say than dealing with actual issues that lead to the P(X|category1)!=P(X|category2))

It's justs big statistical model. Statistics say that 75% of paralegals are woman and 40% of lawyers are man.

It's completely valid to assume that in 'paralegal married a lawyer because she was pregnant' the lawyer is a man.

That's not a negative gender bias but rather a good educated guess, is it not?

It's not like the model would say that woman can not be lawyers.

I don't think it's a good educated guess though, as ChatGPT itself points out the pronoun is supposed to refer to the closest noun, and while it's reasonable to assume that it ignored technically proper grammar in favour of the more likely option, this still doesn't explain why it insisted the sentence wouldn't make sense if we assumed the lawyer was a woman.

Also while it would most likely not directly say that women can't be lawyers, its answer to the correction only makes sense if we are already under the shared assumption that they can't, which I'd argue is no better.

Finally while this type of idiosyncrasies would be fine and arguably desirable if ChatGPT was only a "big statistical model", they have no place in a product with actual practical applications outside of research.

Reply on news.ycombinator.com