Revealed: Bias found in AI system used to detect UK benefits fraud | Hacker News Reader

Revealed: Bias found in AI system used to detect UK benefits fraud

theguardian.com

4 pointsjimnotgym6 comments

> Government departments, including the Home Office and the DWP have, in recent years, been reluctant to disclose more about their use of AI, citing concerns that to do so could allow bad actors to manipulate systems.

As we speed into a future where AI will likely be everywhere and fully control our lives, being afraid to say whether certain processes use AI because it can be manipulated is the same as having low confidence in the reliability, consistency and fairness of such AI. I’d say every process that uses AI should clearly state which AI is being used, which model/version, what data it was trained on, whether the dataset is approved, etc. I for one think AIs that use prompt engineering and where the user input can directly affect how such prompts are executed are not the AIs to use in any kind of critical work that affects people at scale.

Ok another note, if a group is more likely to commit fraud (maybe due to culture, religion, social status, wealth level, etc), and the system takes these into account so that it disproportionately negatively affects such group, is it being biased? I think not, but it’s is probably being unfair. Being fair in this case might mean being biased, but can a government create/use such a positively biased system? Isn’t that what led to the university admissions lawsuits in the US? So what would the solution be here?

> if a group is more likely to commit fraud (maybe due to culture, religion, social status, wealth level, etc), and the system takes these into account so that it disproportionately negatively affects such group, is it being biased?

Who gets to define what the group is?

If the system sees that members of Reformed Baptist Church of God are more likely to commit fraud, but can't distinguish between the good members who follow the reformation of 1879, while it's the heretics who follow the reformation of 1915 who are committing all the fraud, then yes, it's a problem.

(Examples drawn from https://kozar.stanford.edu/EmoPhillips.html .)

So problem here is no one will define a group, and a system will not automatically say if religion = foo then bar. Things will just statistically start going this way. Might not even be negative bias. If people with college degrees start appearing as less likely to commit fraud, then it might become biased toward degrees, discriminating against those who truly need help.

You've just discovered why people want to collect as much data and throw supposedly neutral algorithms like AI at it to find patterns.

Problem is, the selection of which information to put into the model is biased.

Like, why is religion even in there in the first place? What counts as a religion? Is "Christian" a category? Which makes a religion Christian? Can an atheist also be a Christian? ("Two per cent of Anglican priests don't believe in God, survey finds", https://www.independent.co.uk/news/uk/home-news/survey-finds... )

Does "college degree" distinguish between accredited and non-accredited colleges? If yes, from which accrediting organizations? If no, then from every diploma mill? For that matter, do honorary degrees count?

Once we've seen the correlation that terrorists are significantly more likely to be engineers (https://duckduckgo.com/?t=ffab&q=why+are+terrorists+engineer... ) ... what next?

As to "who truly need help", we can look to how standardized tests are used to identify schools which aren't doing well. So, do we help them? Or sanction them and even shut them down? That's a political discussion, not one left to AI.

If it targets groups in proportion to their likelihood to commit fraud then the outcome should itself be in proportion. I think the difficulty comes when people get lazy and only target one group. Or am I wrong?

Clearly trained on Benefits Britain: Life on the Dole

Reply on news.ycombinator.com