Digital dystopia: how algorithms punish the poor
theguardian.com
theguardian.com
This is dangourous journalism. The reporter has clearly taken a stance and is not attempting to open up a nuanced and careful discussion of the factors at play here but is instead insighting fear and mistrust of technical progress in the public sphere.
I can understand the fear and concern around the changing digital landscapes and the potential impact on different parts of society. What's needed is a public discourse about these things that doesn't conflate the issues into one single problem of a "digital dystopia". Are we really talking about a "flawed algorithm", for example, or simply the encoding of a badly designed government process?
Technology _is_ changing how poor people interact with the state and this is an important topic to discuss openly and broadly. We'll never have the quality of discussion that's needed while there are fear mongering reporters like this subverting that conversation to the level of luddism.
This is a case where I believe the topic won't be properly discussed unless someone is willing to take a hard stance on the negative side. Someone has to point out all the potential failings of the system that so many people are pouring their lives into building. Even if that person ends up taking a much harder stance than they actually believe in, and becoming a little too disconnected from reality.
The situations being described in the article are horrifying, and sound an awful lot like what you might expect from software bugs in their early stages. That wouldn't be a problem if there was an appropriate human force behind the systems going online, but we've all seen stakeholders push systems into production before they're ready without adequate support.
You're probably on the right track to call it a "badly designed government process", but I'd wager the human element is what softened the blow from that bureaucracy. It shifts the burden of proof from the case worker to the support seeker. It changes the conversation from "You think you deserve benefits? Let's look over your evidence" to "You've already been rejected, why should we support you?". When you're talking about people surviving, that's a significant difference.
Sure, that will get the issues discussed - discussed and then promptly rejected. A bad arguer arguing for the right side can do a lot of damage. For example, imagine how successful climate change skeptics would be if someone got in to the news by saying that New York was a month away from flooding.
Most often the people who call themselves that aren't even competent enough to explain their own viewpoint.
I'm interested in facts, not opinions completely lacking in (and separated from) facts.
In practice, a biased statement can bring better counterarguments: https://meta.wikimedia.org/wiki/Cunningham%27s_Law
So true! My issue here isn't so much the bias but the poor reporting on what the reporter means by terms like "automating poverty". The difference between algorithms, AI and the organizational processes behind them are significant to the discussion. My view is that tech reporting in the media tends to be so inprecise and unfocused that the quality of discussion suffers as a result.
In meatspace, and even on many parts of the internet, that just isn't true.
If a newspapers advocates for the devil, that's the only side people will hear.
Isn't that the same thing? If you take a flawed algorithm haphazardly implemented through government process and reify it in code, that doesn't fix any of the problems with it, and it may well make the situation worse by removing flexibility or the possibility for human intervention.
These are two distinct things. Understanding how technology exacerbates the problem is one thing. Attacking technology as the source of the problem is another.
So, the underlying problem is the algorithm, not the algorithm? That totally makes sense!
Also, if there are two contributing factors to a problem, and removing either would solve the problem, it is completely useless to arbitrarily declare that one of them is "the real problem".
There's a massive difference between people carrying out directions and machines carrying out the codification of those same directions. The more rigidly codified the system, the shittier life is for nonstandard cases. Machines are the ultimate rigid system.
Consider how much shittier someone's life is when a corner case happens and there's no one to help you resolve it. E.g https://gazette.com/news/born-in-the-usa-without-a-shred-of-...
We can and it's actually happening. So called "dangerous" journalism and "quality" discussion are not mutually exclusive. The article actually mentions one such discussion happening at the level of the UN, an inquiry headed by Special Rapporteur Philip Alston.
In essence, it's about cost cutting. Governments seem to think that machines can replace humans in making decisions where there is nuance involved and the stakes are extremely high. It is frightening and should not be accepted. The furthest the automation should go is to flag irregularities for review. Instead, the machines are given far too much autonomy, with the robodebt collection being a scary example.
You (and Guardian, it seems) are falling for the fallacy of the danger of automation - "it's acceptable for bad decisions to be made as long as it's been decided by a human".
Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?) and benchmark against that - the same way you would measure if a department comprised of humans were doing the same job.
Not necessarily. Even bureaucratic systems tend to recognize that humans make imperfect decisions -- we're susceptible to everything from bribery to exhaustion. These systems then tend to not treat first decisions as final, and they allow an opportunity to appeal.
But what happens when the decision is made by an algorithm, especially one that wasn't built to give an explicable reason?
> Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?)
That's not a good metric.
Suppose the algorithm were in fact perfect and nobody starved -- except you. For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm would be doing a fantastic job compared to an imperfect human system, but it would also be profoundly unjust.
So the problem is not automation, is lack of recourse.
Lack of recourse is a real problem that already exists today. People do go unattended by not knowing how to navigate bureaucratic structures, and not having money to pay a lawyer. It also affects the most the countries the Guardian article targets as "automating poverty".
> For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm
This is not an algorithmic issue.
If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either.
The problem is automation, if it exaggerates the "lack of recourse". It means that, without being careful, adding automation would screw up most of such systems. Thus such articles to boost awareness and make sure automation is done well. I'm disappointed with what a hard time many on HN seem to have with accepting the need to be cautious. Especially with Gov, were you cannot just find or build a better alternative on the open market.
> If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either.
Is that actually the case in the US? The programs (run by the church and heavily supported by german gov) I've come in contact with certainly did not. Never was at a soup kitchen, but would be surprised for them to ask for IDs.
The problem is not the automated systems. The problem is zero recourse.
If nothing was automated then we'd have to wait hours on the phone to get to an actual human to do something trivial. Or make an appointment, travel somewhere, spend few hours to get something simple done.
In an automated world most of the time things work out.
The situation becomes despicable where there's no one to complain to and no way to seek redress.
In the tech world it became pretty common to seek support by Twitter shaming. Often there's no other route.
Still, it's not the automation that is at fault. It's the greedy humans. They could save 80% of the cost while keeping everyone happy but they'll chose to save 90% of the cost at the price of making many miserable.
Can't change the human nature?
However, the human beings who work there could give you advice on where and how to get an ID - which might differ depending on your individual circumstances. A computer probably wouldn't give you any more information than you could read on the internet, and if that info doesn't make sense or doesn't apply to you, you're going to have trouble finding better information.
Automation can be better, more accurate, and in general kinder, but it can also be more profitable, efficient, uncaring and cruel, and when its pointed out the people at fault can avoid blame far far easier.
An example is raising the social security age from 65 to 67. How many lives did that simple change affect? What repercussions did that have on that individual's family? We don't really think about it, we just think about it's effects on the bottom line.
"One death is a tragedy; a million is a statistic" -Joseph Stalin (1879-1953).
- There's no distinction between the different types of systems involved.
- Human problems are blamed on the system (A lack of human support isn't the programs fault)
- Individual stories are used not to underline broader evidence, but meant to be the evidence ("a man died in india, therefore all computers are evil").
- There's constant generalizations. It's not "some algorithms", it's "the algorithms" that are being blamed. (Ctrl+F shows the word "some" appears just once in the text FFS)
I don't know if the author is satisfied with this article; if they are, they should be fired. Misinformation like this is ammunition for those crying "fake news" to further destabilize the political landscape, which, in turn, harms the reputation of real journalism, the kind that our societies rely on to maintain a functioning democracy.
Oh, you can sue programs now? I wasn't aware of that!
I mean ... seriously? What are you even trying to argue there?!
Weapons of Math Destruction https://www.goodreads.com/book/show/28186015-weapons-of-math...
Automating Inequality https://www.goodreads.com/en/book/show/34964830-automating-i...
I'm not sure I understand, would you clarify this for me? It's been my experience that pure reaction is usually bad for discourse, so I'd like to know more.
"we need a measured response instead of a purely reactionary response which will just make things work"
At least, that was my first impression, definitely could be interpreting the comment incorrectly.
> Instead of talking to a caseworker who personally assesses your needs, you now are channeled online where predictive analytics will assign you a future risk score and an algorithm decide your fate.
If the caseworker is racist, this isn't a worse scenario... unless such racism has also been baked into the risk assessment algorithm, which is definitely a possibility. But at least from my personal experience, I trust our ability to identify and de-train racism from assessment networks more than I do our ability to consistently identify and hire non-racist caseworkers. Deprogramming racism out of humans is hard (and even if you succeed for one, you can't copy their state-vector into the brains of their peers).
That means if we try to tune the system to make it less racist, we’ll be making it less accurate. In essence, the system isn’t really racist, it’s a reflection of the racism in society which is leading to these outcomes. Ultimately, the problem is that putting someone in jail increases the likelihood that their relatives will commit crimes. It increases the likelihood that both they and their friends and family will reoffend. It’s a vicious cycle and it doesn’t appear to have any technical solution.
[1] https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
Predictability/accuracy isn't the important part except as means to the end - the outcome.
Another thing about racism in humans is that it's easier to detect than once it's codified in a technological black box.
In a society were it is normalized, sure. But can you imagine what systems such a society would come up with? So far we are only talking about shitty automated systems accidentally producing unwanted outcomes by e.g. reflecting subtle / unwitting / intolerable issues.
> and be much harder to correct once bias is identified
At the end of the day, I'd expect most social workers wanting to help and do a good job at it. AI hasn't reached that level, yet. So I'd expect a much better baseline from humans, with "fixing" outliers being much harder, yes.
It very much resembles the shift from human tech support operators to software-based robo operators. In effect, it’s a wall to keep you away.
The apotheosis of this concept is Google, a company that goes to ridiculous lengths to stop people from being able to contact a human being that works there, unless you’ve paid them money for the privilege.
If you look at modern enterprises, digital technology is most often used in the most un-imaginary way possible. For surveillance of workers. KPI and rationalization are keywords here.
That has made working profoundly worse. Until we see more constructive deployments of information technology, I think the recoil of accusations like this should be more tempered.
And yes, there is a certain subjection to data. In the health care sector, nurses get a fixed amount of minutes to be done with the patient. Based on collected data that allegedly gave us the insight to make that determination.
It should be the other way around. Managers should fear that digital technology surpasses their abilities to efficiently coordinate tasks and give nurses information about the most urgent cases. Digital technology has failed spectacularly to achieve any of that.
See the last few paragraphs.
It is worth noting that this is merely the introduction piece to a series of articles that the Guardian seem to be running this week[1].
It's intention _is_ to be brief.
[1]https://www.theguardian.com/technology/series/automating-pov...
In respect of Aadhar in India, I know the Anti pattern where they deduplicated many undeserving people receiving 'welfare' payouts from the state.
Is this to state that either of these approaches are right?
No. Eliminate the skunk out of the politics, and such things won't happen.
The problem is that Aadhar (introduced initially for subsidies) has become a biometric ID system to link more and more of a person's life to this government ID. A few weeks ago a child died of rabies and the first excuse the hospital gave to delay treatment was that they wanted her Aadhar: https://timesofindia.indiatimes.com/city/agra/rabies-infecte...
I am disappointed whenever I see Bill Gates mindlessly applauding Aadhar. It is easy to side with technocratic ideas from just reading headlines, but if a biometric surveillance state is such a great idea he should have it advocated for it in the US first. (Another example of a technocratic authoritarian move he initially made some positive murmurs about was demonetization, which now, three years later, has proven to have had a disastrous effect on India's economy.)
I agree with this. It’s not acceptable to use the developing world as a staging box for ideas and policies that are, at best, in beta and likely to have bugs.
https://www.livemint.com/Money/aJ6VPyH94gNVwm8alEs87L/11-lak...
So there were roughly 1.1m duplicate tax numbers, and were cut off from writing tax savings of roughly $4000+ a year.
Like I said, it's not a question of what it is, but how it's used that matters.
When people (inside and outside government) treat social security schemes and tax payer money as profiteering schemes, bad things will happen.
There is no surprise here. Every aspect of life will become more automated and reliant on technology. Of course anything involving human safety (healthcare, benefits) has to account for possible problems in the system. There has to be some type of support where you can talk to a human. This is something that should be debated. The author had plenty of provocative information without resorting to this writing style.
>Journalism and activism can’t be separated.
They should and they can. Either have a separate publication, or at the very least have a separate, clearly delineated "Opinion" section. Otherwise you run the risk of media trust becoming very low, and thus media losing its ability to guard the republic (or the democracy).
Consider a similar assertion - "journalism and advertising can't be separated" - it's obviously false. Granted, there is some undue influence from advertisers to be expected. However this is considered a negative that should be avoided. And indeed the good practice is to have a "firewall" between the advertising department, and the news desk, in any publication.
Practically it is impossible to separate the two. Even news reporting doesn’t usually come free of editorializing. There are places where you can go to get ‘just the facts ma’am’ but newspapers have never and will never be those places.
(1) And an author can identify words designed to elicit emotional response and work to minimize them in their writing if they want to focus on the facts vs. their interpretation of the facts.
(2) Listen friend, only an idiot doesn't understand words and phrases have emotional tone, and an author chooses whether they want to piss you off or stay out of your emotions when they choose their words.
... these two sentences have the same conceptual content.
If you show efforts in reporting neutrally, your readers will pick that up.
I agree that media companies, which are part of a large multi-billion industry, have their shortcomings. That makes trying to keep an open mind even more important for journalists. There is a reason why being independent is still a positive qualifier for journalists.
"In Illinois, the Guardian has found that state and federal governments have joined forces to demand that welfare recipients repay “overpayments” stretching back in some cases 30 years."
This isn't technology, it's politicians and bureaucrats deciding policies. There is no reason the technology can't write off aged debts like this.
The whole article stinks.
The underlying argument of critics of algorithms that make decisions, in government and in private companies, is that this technological apparatus becomes a justification in itself: if a mistake is made, blame the algorithm; if the algorithm says so, it must be right — we can't make exceptions.
Over time, people and institutions begin to shape their own actions so they don't fall afoul of the criteria that can be reverse-engineered out of a black box decision-making system.
Regarding "this isn't technology", this doesn't invalidate the criticisms: part of the argument is that, indeed, what gets called "technology" often shouldn't be simply described as technical tools and processes. They're often the opinions and goals of individuals (politicians, bureaucrats, and other wielders of power) transformed into code, data and other technical artifacts; sometimes this transformation merely hides the underlying sources of these opinions and goals.
watch it