Cathy argues that the use of algorithm in some contexts permits a new scale of harmful and unaccountable systems that ought to be reigned in.
https://www.penguinrandomhouse.com/books/241363/weapons-of-m...
"A computer can never be held accountable, therefore a computer must never make a Management Decision." IBM presentation, 1979
Hence coders are the new managers, managers just funnel the money around, a job which can be automated
IBM in 1979 was not doing anything different to 2024. They were just more relevant
= Presentation, 21st Century
A computer is not alive. A computer system is a tool that can do harm. It can be disconnected or unplugged like any tool in a machine shop that begins to do harm or damage. But a tool is not responsible. Only people are responsible. Accountability is anchored in reality by personal cost.
= Notes
Management calculates the cost of not unplugging the computer that is doing harm. Management often calculates that it is possible to pay the monetary cost for the harm done.
People in management will abdicate personal responsibility. People try to avoid paying personal cost.
We often hold people accountable by forcing them to give back (e.g. community service, monetary fines, return of property), by sacrificing their reputation in one or more domains, by putting them in jail (they pay with their time), or in some societies, by putting them to death ("pay" with their lives).
Accountability is anchored in reality by personal cost.
To really foul things up requires scalability
The original point seemed to me to be "we can't use computers because they're not accountable". I say, we can, because we can do fault analysis and fix what is wrong. I won't say "we can hold them accountable", to avoid the category error.
If your algorithm kills someone, is the accountability an improvement to the algorithm? A fine and no change to the algorithm? Imprisonment for related humans? Dissolution of some legal entity?
An algorithm has no concept of consequences (unless programmed to be aware of such), and the more plausibly whoever wrote it can deny knowledge of the resulting consequences, also the more whoever wrote it can avoid consequences/accountability themselves. After all, we can tell Soldiers or Clerks that ‘just following orders’ is no excuse. But computers don’t do anything but follow orders.
Most people/organizations/etc have strong incentives to be able to avoid negative consequences, regardless of their actions or the results of their actions.
Everyone around them has strong incentives to ensure negative consequences for actions with foreseeable negative outcomes are applied to them.
Sometimes, organizations and people will find a way for the consequences of their actions to be borne by other people that have no actual control or ability to change actions being performed (scapegoat). Accountability ideally should not refer to that situation, but sometimes is abused to mean that.
That tends to result in particularly nasty outcomes.
What I read is yes, the point is revenge. If I can offer you a different way of preventing harmful activity, apparently you're not interested. There has to be some unpleasant consequences inflicted, you insist on it.
I think you should reconsider.
Ideally, the stick never gets used. We aren’t dealing with ideals, however, we have to deal with reality.
On any sufficiently large scale, an inability/lack of will to use the stick, results in wide scale malfeasance. Because other constrains elsewhere result in wide scale push to break those rules/boundaries/structures for competitive reasons.
No carrot, magnifies the need to use the stick, eh? And turns it into nothing but beatings. Which is not sustainable either.
It has nothing to do with revenge. But if it makes you feel more comfortable, go ahead and call it that.
It’s ensuring cause and effect get coupled usefully. And is necessary for proper conditioning, and learning. One cannot learn properly if there is no ‘failure’ consequence correct?
All you need to do to verify this is, literally, look around at the structures you see everywhere, and what happens when they are or are not enforced. (Aka accountability vs a lack of it).
Suppose I’m a bad actor that creates an unfair algorithm that overcharges the clients of my company. Eventually it’s discovered. The algorithm could be fixed, the servers decommissioned, whatever, but I’ve already won. If the people who requested the algorithm be made in that way, if the people who implemented it or ran it see no consequences, there’s absolutely nothing preventing me from doing the same thing another time, elsewhere.
Punishment for fraud seems sane, regardless of whether it’s enabled by code or me cooking some books by hand.
The evolutionary function certainly encourages it, correct?
Ignoring that means that not applying consequences makes one actually culpable in the bad behavior occurring.
Especially if nothing changed re: rules or enforcement, etc.
I don't see what the problem is. There's malice, there's negligence, and there's accident. We can figure out which it was, and act accordingly. Must we collapse these to a single situation with a single solution?
Sure! But oop mentioned a phenomenon that some companies can hide after algorithms and reject taking responsibility for it. If machines cause damage, people can easily find who's fault, but sometimes the same way does not work on software.
Algorithms are used by people. An algorithm only allows "harmful and unaccountable systems" if people, as the agents imposing accountability, choose to not hold the people acting by way of the algorithm accountable on the basis of the use of the algorithm, but...that really has nothing to do with the algorithm. If you swapped in a specially-designated ritual sceptre for the algorithm in that sentence (or, perhaps more familiarly, allowed "status as a police officer" to confer both formal immunity from most civil liability and practical immunity from criminal prosecution for most harms done in that role), it functions exactly the same way: what enables harmful and unaccountable systems is when humans choose not to hold other humans accountable for harms, on whatever basis.
"The Unaccountability Machine," based on Mandy's summary in the OP, argues that organizations can become "accountability sinks" which make it impossible for anyone to be held accountable for problems those organizations cause. Put another way (from the perspective of their customers), they eliminate any recourse for problems arising from the organization which ought to in theory be able to address, but can't because of the form and function of the organization.
"Weapons of Math Destruction" argues that the scale of algorithmic systems often means that when harms arise, those harms happen to a lot of people. Cathy argues this scale itself necessitates treating these algorithmic systems differently because of their disproportionate possibility for harm.
Together, you can get big harmful algorithmic systems, able to operate at scale which would be impossible without technology, which exist in organizations that act as accountability sinks. So you get mass harm with no recourse to address it.
This is what I meant by the two pieces being complementary to each other.