Accountability sinks

I remember experiencing this in one of the German airports/airlines and having that exact thought.

It was this fully automated airport, where the checkin is self serviced and you only interact with computers.

Eventually, when I inserted my boarding pass I had a printed piece of paper back that said that they had to change my seat from aisle to midseat

I then tried to find someone to talk to the entire way, but computers can only interact in the way the UI was designed, and no programmer accounted or cared for my scenario

The ground attendant couldn't have done anything of course because it wasn't part of the scope of her job, and this was the part of germany where nice was not one of their stereotypes.

Eventually I got a survey a week later about a different leg of the flight, so could I really complain there? that one was fine? I had a paranoid wonder if that was intentional

I've long thought that that is one of the main functions of corporations. There's a reason they're called limited liability. The fact that you can conjure up new companies at a whim makes it easy to shuffle responsibility into an obscure corner.

This is a strong reason that corporations should not be considered people. People are long-lived entities with accountability and you can't just create or destroy them at will.

Cathy O'Neil's "Weapons of Math Destruction" (2016, Penguin Random House) is a good companion to this concept, covering the "accountability sink" from the other side of those constructing or overseeing systems.

Cathy argues that the use of algorithm in some contexts permits a new scale of harmful and unaccountable systems that ought to be reigned in.

https://www.penguinrandomhouse.com/books/241363/weapons-of-m...

This is a terrible example because it's simply not true:

> Davies gives the example of the case of Dominion Systems vs Fox News, in which Fox News repeatedly spread false stories about the election. No one at Fox seems to have explicitly made a decision to lie about voting machines; rather, there was an implicit understanding that they had to do whatever it took to keep their audience numbers up.

Rupert Murdoch conceded under oath that "Fox endorsed at times this false notion of a stolen election."[1] He knew the claims were false and decided not to direct the network to speak about it otherwise.

Communications from within Fox, by hosts, show they knew what they were saying was false.[2]

These two examples clearly fit the definition of lying [3].

The "External Links" section of Wikipedia gives references to the actual court documents that go into detail of who said what and knew what when [4]. There are many more instances which demonstrate that, indeed, people made explicit decisions to lie.

[1] https://www.npr.org/2023/02/28/1159819849/fox-news-dominion-...

[2] https://www.nbcnews.com/politics/elections/dominion-releases...

[3] https://www.dictionary.com/browse/lie

[4] https://en.m.wikipedia.org/wiki/Dominion_Voting_Systems_v._F...

My suspicion I'd that one of the major appeals of automation and especially "app-ification" for management and C-Suite types is specifically its ability to break accountability links.

A lot of corporations now seem to have a structure where the org chart contains the following pattern:

- a "management layer" (or several of them) which consists of product managers, software developers, ops people, etc. The main task of this group is to maintain and implement new features for the "software layer", i.e. the company's in-house IT infrastructure.

Working here feels very much like working in a tech company.

- a "software layer": This part is fully automated and consists of a massive software and hardware infrastructure that runs the day-to-day business of the company. The software layer has "interfaces" in the shape of specialized apps or devices that monitor and control the people in the "worker's layer".

- a "worker's layer": This group is fully human again. It consists of low-paid, frequently changing staff who perform most of the actual physical work that the business requires (and that can't be automated away yet) - think Uber drivers, delivery drivers, Amazon warehouse workers, etc.

They have no contact at all with the management layer and little contact, if any, with human higher-ups. They get almost all their instructions through the apps and other interfaces of the software layer. Companies frequently dispute that those people technically belong to the company at all.

Whether or not those people are classified as employees, the important point (from the management's POV) is that the software layer serves as a sort of "accountability firewall" between the other two layers.

Management only gives the high-level goal of how the software should perform, but the actual day-to-day interaction with the workers is exclusively done by the software itself.

The result is that any complaints from the worker's layer cannot go up past the software - and any exploitative behavior towards the workers can be chalked up as an unfortunate software error.

That's a really thought provoking article. And my thinking is this highlights the importance of government consumer protection agency/laws as the protection against that. I.e. when you fly through Europe or use European airlines, there is this EU law that gives you compensation of ~ 600 EUR if your flight is delayed by more than 3 hours or cancelled or whatever. This is a good insurance that no matter what BS is thrown at you at the airport by the company, you will get your compensation. And the process of getting the money is reasonably straightforward. What that gives is a way of avoiding any kind of airline systems, and just leads to the compensation. Also I hope that serves as an actual motivation for the airline to perform reasonably well, because otherwise they'll pay too much in fines. I think we really need this kind of protection laws in order to avoid the situation of chatbot-wall shielding companies from customers.

I experience this pretty often with the newfangled, automated government e-filing systems.

As a screen-reader-using person who cannot use pen and paper without assistance, I was once quite enamored by them, but I've changed my stance a bit.

The thing about pen and paper is that it accepts anything you put in, and it's up to a human to validate whether what you put in makes any sense. Computers aren't like that, if they tell you that the numbers in your application have to match up, you need to lie to the government to make them match up, even if you're a weird edge case where the numbers should, in fact, be slightly off and "inconsistent" with each other.

I called the local govt office responsible for this specific program, and they essentially told me to lie to the government in not so many words. Their system is centrally managed, they have no power of introducing updates to it, they wish they could fix it, but even they aren't empowered to do so.

When I was a grad student in STS I was considering doing a project on how software can function as an "agency adjuster" where individuals come to bear the risks of something (generally an economic transaction) and the majority of the profits go to the owner of the software. In many ways Uber & related services are about allowing individuals to take on very low-probability high-acuity downside risk for a small fee.

>The comparisons to AI are obvious, in as much as delegating decisions to an algorithm is a convenient way to construct a sink.

There is a flag on my LinkedIn account that bars me from getting a "follow-me" link on my profile.

No one of their support team knows why. No one knows since when. No one knows when it will change.

We are already living in this world.

Organizations exist to remove moral culpability

Judge, Jury and Executioner Firing Squad Limited Liability Organisation

Humans like to sleep at night. An emergent property of our rule of law is that it exists in a way to reduce the moral culpability of any individual. A police man, a jury member, a judge, a inspector, an executioner, a jailer, they all exist in very neat boxes. These boxes allow them to sleep at night. Surely the Judge has few qualms going by the recommended mandatory minimum, after the jury, who is assured the judge will provide a fair sentence, and the executioner doubly so, with double the potential moral hazard, is certain at least two other parties have done their due diligence.

these systems prevent a single actor from acting. More like they allow a series of hand offs, so by the time the jailer is slamming the doors shut, they are bereft of any investment in the morality of the outcome

The firing squad, with seven guns, all line up, with just one loaded. The rest are blanks. Each man can sleep at night, regardless if the murdered man was surely deserving of death

large institutions, organizations and objects are scale are fully inhumane

I would rather have my jailer be my judge and my executioner be each man or woman on the jury. Isolating each of these things allows the individuals to have almost a powerless notion of 'completing our task'. As if all tasks completed would add up to a moral outcome

Should juries be formed to perform the whipping of an individual, the institutionalization in their own homes, the judge forced to starve a prisoner in his cell, i find the outcomes would be different

I was thinking about something similar today. Sometimes accountability can be a blocker, for example for hiring.

If you have 1 candidate, it's an easy call, if you have 3 candidates, you evaluate in less than a week. If you have 200 candidates, you need to hire somebody to sift through the resumes, have like 5 rounds on interview and everybody chiming in, whoever pulls the trigger or recommends someone is now on the hook for their performance.

You can't evaluate all the information and make an informed decision, the optimal strategy is to flip a 100 sided die, but no one is going to be on the hook for that.

Too focused on the bottom level. If a given business process results in employee A doing their job correctly according to the process, passing work to employee B doing their job correctly according to the process, passing work to employee C doing their job correctly according to the process, and the end result is shit, then the person who is accountable for the end result being shit is the manager who is responsible for the process itself. As more and more employees are involved, and the processes get more and more hierarchical (rather than "employee A", you have "middle-manager M"), then the person with accountability is higher and higher up the hierarchy, who also has more and more power and responsibility to fix it.

The idea of "unaccountable" failures only makes sense if both (a) the problem is so systemic that actually an executive is accountable, (b) the executive is so far removed in the hierarchy from the line employees doing the work that nobody knows each other or sometimes even sits on the same campus, (c) the levers available to the executive to fix the problem are insufficient for fixing the problem, e.g. the underlying root cause is a culture problem, but culture is determined by who you hire, fire, and promote, while hiring and firing are handled by "outside" HR who are unaccountable to the executive who is supposedly accountable. But really this is another way of saying that accountability is simply another level higher, i.e. it is the CEO who is accountable since both the executive and HR are accountable to the CEO.

No, you have to have an astoundingly large organization (like government) to really have unaccountability sinks, where Congress pass laws with explicit intent for some desired outcome, but after passing through 14 committees and working groups the real-language policy has been distorted to produce the exact opposite effect, like a great big game of telephone, one defined by everyone trying to de-risk, because the only genuine shared culture across large organizations is de-risking, and it is simply not possible to actually put in place both policy and real-life changes to hiring, firing, and promotion practices in the public sector to start to take more risks, because at the end of the day, even the politicians in Congress are trying to de-risk, and civil servants burning taxpayer money on riskier schemes is not politically popular, though maybe it should be, considering the costs of de-risked culture.

This article seems to redefine the word "accountability". In the first sentence:

> In The Unaccountability Machine, Dan Davies argues that organizations form “accountability sinks,” structures that absorb or obscure the consequences of a decision such that no one can be held directly accountable for it.

Why not just call it "no-consequence sinks"?

It's somewhat of an oxymoron to say "accountability" isn't working because there's no consequence. Without any consequence there is no accountability. So why call it accountability in the first place?

This article is describing something along the lines of "shared accountability" which, in project management, is a well known phenomenon: if multiple people are accountable for something, then no one is accountable.

If someone is accountable for something that they can't do fully themselves, they are still accountable for setting up systems (maybe even people to help) to scale their ability to remain accountable for the thing.

Douglas Adams was here in 1982 with the invention of the SEP field

‘An SEP is something we can't see, or don't see, or our brain doesn't let us see, because we think that it's somebody else's problem. That’s what SEP means. Somebody Else’s Problem. The brain just edits it out, it's like a blind spot.

The narration then explains:

The Somebody Else's Problem field... relies on people's natural predisposition not to see anything they don't want to, weren't expecting, or can't explain. If Effrafax had painted the mountain pink and erected a cheap and simple Somebody Else’s Problem field on it, then people would have walked past the mountain, round it, even over it, and simply never have noticed that the thing was there.’

https://en.wikipedia.org/wiki/Somebody_else's_problem

I rather think accountability improved a lot. Esp. with the decline of buerocratic walls.

Accountibility always was down. Back in aristocracy you were never allowed to ask for support. Only in modern civilisation this improved. Middle management, the clueless in the Gervais principle, need their walls.

Don't be fooled by the decline of customer support in big orgs, like Google, Apple, or Amazon. They believe that support cannot scale, or if it's really needed, it needs to be outsourced to India or East Asia.

I feel like the article, or perhaps just the example, is missing the point.

>> a higher up at a hospitality company decides to reduce the size of its cleaning staff, because it improves the numbers on a balance sheet somewhere. Later, you are trying to check into a room, but it’s not ready and the clerk can’t tell you when it will be; they can offer a voucher, but what you need is a room.

This reads from the perspective of a person checking in. But it should read from the perspective of the person who made the decision.

The decision was made like this; On most days we have too many cleaners. If we reduce the cleaners we reduce expenses by x.

On some days some customers will need to wait to check-in. Let's move checkin time from 1pm to 2pm (now in some cases to 4pm) to compensate. n% of customers arrive after 4pm anyway. We start cleaning early, so chances are we can accommodate early checkin where necessary.

Where there's no room available before 4pm, some % will complain. Most of those will be placated with a voucher [1] which cost us nothing.

Some small fraction will declare "they'll never use us again". Some will (for reasons) but we'll lose a few.

But the savings outweigh the lost business. Put some of the savings into marketing and sales will go up. Costs remain lower. More profit.

There is perfect accountability of this plan - the board watches to see if profits go up. They don't care about an individual guest with individual problems. The goal of the business is not to "make everyone happy". It's to "make enough people happy" to keep profits.

[1] the existance of the voucher proves this possibility was accounted for.

So accountability in this case is working - except for the customer who didn't get what they want. The customer feels frustrated, so from their perspective there's a failure. But there are other perspectives in play. And they are working as designed.

In my country they enacted this system for student management that is national.

It handles signups, restauration and housing services, grades, everything.

One example is that the grades are entered by professors and mistakes happen all the time, for everyone, due to the insane server load.

There's no one to complain to, because the excuse is always "it's the system, not us"

Tom Schelling's 'The Strategy of Conflict' touches on similar themes, but mostly in a more positive light.

One of his examples is that you should make yourself unavailable for contact, when you suspect someone is trying to blackmail you.

That's exactly the same severing of a link as described in the article.

Interesting. Wonder sometimes how much of consulting business is motivated by accountability avoidance - "accountability sinks" for hire

>> In The Unaccountability Machine, Dan Davies argues that organizations form “accountability sinks,” structures that absorb or obscure the consequences of a decision such that no one can be held directly accountable for it.

Government and Civil Servant are the biggest example. I guess its time to re-watch "Yes Minister".

patio11 has an episode of his podcast Complex Systems where he interviews the author of this book. It’s well worth a listen / read, as is the book itself. https://www.complexsystemspodcast.com/episodes/dan-davies-or...

Taleb's Skin in the Game seem to be related to this, but from a different optic. Goodhart's Law is also mentioned, but is not the core argument. In the end, is about agency, who have it, and system dynamics to get rid of responsibility.

There was a leaked memo essentially instructing to form a committee when you make an illegal decision so that one person cannot be sent to jail. Does anyone remember this? I've had a hard time finding it

you can't have accountability in a world where you can be a good family member and work for a company that manufactures bombs that kill families. This doesnt compute in humans. They cant deal with that. If you cant handle accountability for that ... who cares about you getting into your hotel room a little later or missing a flight or not getting health insurance ... no biggie.

what you can have is a discussion about this or a blog post that is read by people and maybe some new subscribers, so no worries - all is not lost. :)

Accountability sinks sounds a lot like the Toyota factory story, where on the contrary, every employee in the factory could pull the 'stop' lever if they thought there was a quality problem. Which of course drastically increased quality and feedback because the process is interrupted and stops.

But I don't think it is quite so black and white in the world. Because the legal system is also a way to give feedback to companies. And it can stop them in their tracks.

I liked this article a lot - it made me think about the ways large companies operate from a different viewpoint.

At the same time, though, I think it's a mistake to leave out the fact that, in many ways, modern society is just so fundamentally complex that we (as a society at large) deliberately forego demanding accountability because we believe the system is so complex that it's impossible to assign blame to a single person.

For example, given this is HN and many of us are software developers, how many times have we collectively supported "blameless cultures" when it comes to identifying and fixing software defects. We do this because we believe that software is so complex, and "to err is human", that it would be a disservice to assign blame to an individual - we say instead that the process should assume mistakes are inevitable, and then improve the process to find those mistakes earlier in the software lifecycle.

But while I believe a "blameless culture" is valuable, I think a lot of times you can identify who was at fault. I mean, somebody at CrowdStrike decided to push a data update, without verifying it first, that bluescreened a good portion of the world's Windows machines and caused billions in damages.

I just think that if you believe "accountability sinks" are always a bad thing, don't forget the flip side: would things always be better if we could always assign "root cause blame" to a specific individual?

Systems Theory would describe this as "intrinsic responsibility".

From Donella Meadows: “Intrinsic responsibility” means that the system is designed to send feedback about the consequences of decision-making directly and quickly and compellingly to the decision-makers.

In the 400 blows, Truffaut plays a schoolteacher whose disciplinary methods in the classroom only accentuate the rebelliousness of a boy. At home, this trickles down and he decides to run and drink milk. At the end, we find him on a beach and the film ends.

Enforcing copyright law through an honest projection of 35mm film footage is a philanthropic endeavour. Making sure that every member of the production team, even the gaffers and stage hands take part in the exclusivity of re-capitalisation efforts, like the Fox complaint, is purely, legalistic sleight of hand.

The future iterations of this are purely terrifying. This is so elegantly demonstrated by the Oscar Nominated short film “Please Hold” from 2022. Picked up by mistake by a roving police drone, a young man is incarcerated autonomously and has no way of release outside of money or time.

https://m.imdb.com/title/tt11383280/?lang=en&ref_=ext_shr_ln...

I think this is related to the concept of “aligned incentives” in a way.

The chain breaks when incentives aren’t aligned and there’s a cascade of crap that roles downhill that seem like bad decisions. When, in fact, as the article points out, the decision made didn’t take knock on effects into consideration.

I’ve learned that seemingly poor or even terrible decisions almost always make sense in the context of when and where the decision was made.

What the article describes seems like a parallel concept (and an important one.) I wouldn't call them Accountability Sinks, though, as they seem more like Accountability Avoiders. Here are things we might think of as sinks in the real world:

- "Sin Eaters"

- Corporations, especially companies that are spun off and take on all the debt of the original company

- Voluntary stool pigeons (in criminal organizations, etc.)

- Certain religious martyrs

This is a small part of business reality. It's not clear how calling out a specific aspect really helps; indeed, it may hurt when business leaders learn how to do this more effectively. It's not at all clear that calling people irresponsible or unaccountable is actually effective at changing the transaction features; it likely makes things worse.

Transaction cost economics since the 1960's has been enumerating aspects like these, and showing they in fact determine the shape of business organizations and markets. Exported costs (implied in accountability sink) are mostly the rule rather than the exception.

What to do with them? A primary TCE finding is that if there were no transaction costs to adjudicating liability, it wouldn't matter from the social cost perspective where the liability lay (with the perpetrator or the victim) because they would adjudicate it down to their mitigation costs. As a result, the main policy goal for assigning liability (if you want to minimize the total cost to society) is actually to minimize and correct for adjudication transaction costs. (hence, no-fault divorce and car insurance)

The same dynamics are at play in the market and within organizations.

As a participant if your goal is your own profit, you can gain by making it harder to adjudicate and reducing the benefits thereof (hence binding arbitration, waivers, and lack of effective feedback). Doing so is becoming much simpler as virtual transaction interfaces and remote (even foreign) support afforded by software replace face-to-face interactions bound by social convention.

And who wouldn't want to? If you're head of customer support or developer relations, would you document your bugs or face the wrath of customers for things which can't change fast enough? You'd want to protect yourself and your staff from all the negativity. Indeed, with fixed salaries, your only way of improving your lot is to make your job easier.

To me the solution is to identify when incorporating the feedback actually benefits the participants. There, too, the scalability of virtualized software interfaces can help, e.g., the phone tree that automates simple stuff most people need and vectors complex questions to real people who aren't so harried, or the departing-customer survey querying whether it was price, quality, or service that drove one away.

You have to make accountability profitable.

I've observed a phenomenon in corporate accountability resembling quantum behavior:

1. Macro level: Departments claim broad accountability.

2. Micro level: Pinpointing task ownership causes accountability to vanish.

3. Indefinite states: Ambiguous tasks linger without resolution.

4. Entanglement: Dependent tasks inherit this ambiguity.

This creates a system where responsibility exists in superposition, tasks remain unresolved, and accountability becomes increasingly delocalized.

When stock you've held for years rockets to the moon, and your major banking institution with 100 billion dollar annual revenue experiences "technical difficulties" at the login page. You can't sell your meagre crumbs until the spaceship has completed its orbit and you're left with no doubt about who the system doesn't work for.

This hit home and very recent experiences. Was even in my little notebook of ideas to write about.... seems its already been done. Nice job... and you can almost paint all corps. with this brush, esp. decisions made by upper management that is incentivized on financial metrics that can be gamed....

It's a reasonable thing to do. An eye for an eye leaves the whole world blind. If you could demand responsibility from any grievance you have from anyone they could dement responsibility from you for the distress you caused them with your grievance.

My theory is all developed societies converge on having no accountability in the governing positions. Of course, I may and most like am wrong but if you look at say politics you must at least think about this being a real possibility

There has been a lot of talk of silos in the company I work for and the need to break them down. This looks like it could be a big part of why they have been so hard to tackle.

It's in the name, limited liability company.

closely related but in reverse: subsidiarity.

Yawn. Isn't that exactly what https://en.wikipedia.org/wiki/The_Corporation_(2003_film) & https://thecorporation.com/film/ was about twentyone years ago? (Time flies!)

As bottom line at least? Can't have the one without the other, just different aspects of the same shit.

ahem at least in the Western world corporations were invented by the Church, an entity well known to me not accountable....look at the 94 apologies and the Pope's words before and after each apology....

By the way this is what bitcoin was set up to solve...notice it not being solved.

Yeah, but when you post the linkedin profile of the person doing the thing, people call you a scumbag. It's a hard social norm to move the dial on.

I find that the word "accountability" almost always obscures what's being talked about. If we remove it, we can instead talk about understanding and feedback:

As organizations become more complex, it's difficult to understand the consequences of many high-level decisions. Unless great effort is made to gather feedback, it won't happen.

Not only that: the lack of immediate, human communication results in one-way feedback mechanisms, like suggestion boxes and surveys. Many companies clearly want to make this work, because we're constantly prompted and sometimes paid to fill out surveys. But the result is survey fatigue.

The person giving feedback needs to be reassured (by people, not machines) that their feedback matters, or they won't be bothered to do it. Often, it's socially awkward to give negative feedback, so people don't. And often, the employees directly on the scene have incentive to encourage customers to avoid negativity when they fill out surveys.

One way to show that feedback matters is to respond to complaints with some sort of assistance. In the example in the article, that's a voucher. Perhaps somewhere in the organization, that voucher counts as a cost, but it's pretty unsatisfying.

In some organizations, managers are encouraged to work at the support desk occasionally as a more immediate way to understand what's going on. (I remember reading about how Craig Newmark would do this for his website.)