Hacker News Reader

Comment by _Algernon_ | Hacker News Reader

The other side of this coin is that there is an incentive for decision makers to use computers, precisely to not be held accountable. This is captured pretty well by this quote by Neil Postman in Technopoly:

>[B]ureaucrats can be expected to embrace a technology that helps to create the illusion that decisions are not under their control. Because of its seeming intelligence and impartiality, a computer has an almost magical tendency to direct attention away from the people in charge of bureaucratic functions and toward itself, as if the computer were the true source of authority. A bureaucrat armed with a computer is the unacknowledged legislator of our age, and a terrible burden to bear. We cannot dismiss the possibility that, if Adolf Eichmann had been able to say that it was not he but a battery of computers that directed the Jews to the appropriate crematoria, he might never have been asked to answer for his actions.

How does one counteract that self-serving incentive? Doesn't seem like we've found a good way considering we seem to be spearheading straight into techno-feudalism.

View on HN

> How does one counteract that self-serving incentive? Doesn't seem like we've found a good way considering we seem to be spearheading straight into techno-feudalism.

The EU is trying right now. Discussed right now over here:

https://news.ycombinator.com/item?id=42916849

See: https://artificialintelligenceact.eu/chapter/1/ and https://ec.europa.eu/commission/presscorner/detail/en/qanda_...

AI systems are defined[0] in the Act in such a way as to capture the kind of "hands-off" decision-making systems where everyone involved could plead ignorance and put the blame on the system working in mysterious ways; it then proceeds to straight up ban a whole class of such systems, and classifies some of the rest as "high risk", to be subject to extra limitations and oversight.

This is nowhere near 100% a solution, but at least in the limited areas, it sets the right tone: it's unacceptable to have automated systems observing and passing judgement on people, based on mysterious criteria that "emerged" in training. Whatever automation is permitted in these context, is basically forced to be straightforward enough that you could trace back from the system's recommendation to specific rules that were executed and to people that put them in.

[0] - https://artificialintelligenceact.eu/article/3/

The EU AI Regulations has some very broad carve outs added for Law Enforcement, such as real-time facial recognition. And individual EU states still have the authority to carve out exceptions as they wish.

Like every other form of tech regulation done by the EU, it's all bark but no bite because politicians of individual countries have more power than MEPs.

What's to stop Fidesz, PiS (if they return to power), etc from carving out broad exceptions for their own Interior Ministries? They've already done this with spyware like Pegasus.

Instead of techno-feudalism, it's basically techno-paternalism which is essentially the same thing. In both cases, individual agency is being limited by someone else.

At least someone is trying. I don't see anyone doing better anywhere else in the world.

A half assed effort is worse than no effort, because a half assed attempt can easily be weaponized

The issue is a lot of EU legislation becomes powerless, as it's up to individual nations to implement rulings or define exceptions.

And in plenty of cases, these exceptions tend to be broad and equally rife to lobbying like those you'd see in the US.

And there is A LOT of lobbying happening at the nation state level (the level that actually matters) to carve out exceptions.

I already pointed this out with spyware, and there's no reason to presume the exact same thing won't happen with weaponizing AI/ML tooling.

In many ways I share your pessimism, but let me challenge you:

> A half assed effort is worse than no effort, because a half assed attempt can easily be weaponized

This part I'm not following. I don't see the EU effort as half-assed so much as just lacking teeth due to realpolitik. That's unfortunate but still a step in the right direction. What would you propose as an alternative? If you're just saying "we're fucked" that's a point of view, but it doesn't explain how the current attempts will be weaponized. All else being equal, I'd argue that having these issues in the global discussion is better than not. If they're not then how do they get addressed?

Somehow, I think that using a bureaucracy to fix this bureaucratic tendency won’t solve it. Not that it’s not well-intentioned, but at some point we need to expect that individual humans to behave ethically, or else we’re doomed. Over and over, we try to fix society by adding more and more bureaucracy and the situation only gets worse.

We keep adding layers of bureaucracy precisely because we can't expect individual humans to behave ethically at scale.

Personal ethics aren't fully personal in nature - they're just as much a function of social environment as of one's "heart of hearts". Ethical behavior being expected and respected makes it easier for any individual to stick to their principles in the face of conflicting factors, like desires or basic self-preservation instinct.

Individual ethics in social context is a powerful organizer, and can keep small groups of people (30-50) stable and working together for a common goal. Humans evolved to cooperate together at that size - such groups can self-govern and self-police just by everyone's sense of right and wrong. But it quickly breaks down past that size. Letting some individuals assume leadership role and direct others only stabilizes the group up to ~150 people (Dunbar number); past that, the group loses cohesion and ends up splitting up.

Long story short: larger groups have survival advantage both against environment and over smaller groups. Because of it, we've eventually learned how to scale human societies to arbitrary sizes. The trick to do it is hierarchical governance to overcome Dunbar limit, and explicit rules to substitute for intimate relationships. By stacking layer upon layer, we grew from tribes of 150 governed by the spoken words of their chiefs, through kings managing networks of nobles via mutual obligation, to modern nation states that encompass millions of citizens in a hierarchy that's 4+ level deep (central government -> provinces -> regions -> towns), slowly building up more levels as nation states group into blocs. With each layer we added, complexity of explicit governance grew, giving rise to what we now call bureaucracy.

The modern bureaucracy isn't some malignant growth or unfortunate side effect - it's the very glue that allows human societies to scale up to millions of people and more. It's the network of veins and nerves of a modern society - it's what allows me, one tiny cell, to benefit from the contribution of other cells and contribute on my own, and it's what keeps all the cells working as a whole. This means that, yes, as the society grows and faces new challenges, the solution usually really is more bureaucracy.

The defense against this is to have very clear legal principles that identify the person or people fully accountable for the machine's decisions.

Admittedly this may result in some strange results if followed to its logical conclusion, like a product manager at a self-driving car company being the recipient of ten thousand traffic tickets.

>identify the person or people fully accountable for the machine's decisions.

Which required more tech, not less.

Not necessarily.

Tbh I’m in favor of holding C-suite responsible for the actions of their company, unless the company has extremely clear bylaws regarding accountability.

If, say, a health insurance provider was using an entirely automated claim review process that falsely denies claims, I think the C-level people should be responsible.

> product manager at a self-driving car company being the recipient of ten thousand traffic tickets.

I'm the case of self driving cars, the company itself could be held liable. Everyone invested in the company who puts a bad product in the market should be financially impacted. The oligarchy we are heading for wants no accountability or oversight -- all profit, no penalty.

> The defense against this is to have very clear legal principles that identify the person or people fully accountable for the machine's decisions.

Most legal principles are designed to reduce liability. That's the whole point of incorporation, for example.

Incorporation is to reduce liability for debt. It's not supposed to reduce liability for criminal negligence. Or other criminal offences.

At least in the US, there's precious little difference. If a company is found guilty of a crime, the C-suite isn't thrown in prison; they pay a fine.

If a company is found to have committed a tort against a party, they pay damages.

There are exceptions (the Volkswagen diesel scandal comes to mind) but generally both punishments entail paying out a monetary amount that is often lower than the profit generated by the crime, often because of tort reform or because of fine amounts that are out-of-date with current corporate revenues.

There was one banker imprisoned for the 2008 financial crisis, but he was hardly responsible.

No, legal principles are designed to specify liability.

This. I'd go as far as to say that the law mostly tries to conserve liability, in the "energy conservation" sense. Once harm is defined and quantified, the consequences have to be discharged somewhere, and there's tons of rules that try to sensibly distribute them among parties involved, while counteracting everyones' attempts at diffusing liability or redirecting it somewhere else.

On that note, after some time working in cybersec and GRC fields, I realized that cybersecurity is best understood in terms of liability management. This is what all the security framework certification and auditing is about, and this is a big reason security today is more about buying services from the right vendors and less about the hard tech stuff. Preventing a hack is hard. Making it so you aren't liable for the consequences is easier - and it looks like a network of companies interlinked with contracts that shift liability around. It's a kind of distributed meta-insurance (that also involves actual insurance, too).

> (that also involves actual insurance, too)

My eyes were opened to this when management wasn't just talking about deleting unneeded private data just as the right thing to do, but specifically how it could reduce our insurance premiums.

In business, everything that can be ultimately always gets reduced to money. And in vast majority of cases, for ongoing business operations and events that don't directly hurt or kill people, it actually turns out fine.

The core insight that made me start to understand how companies see the world, was reading about ocean freight shipping. Specifically, what happens when there's a bad weather, or dangerous malfunction, or other such unexpected event, that throws some containers off the ship or forces the sailors to intentionally dump them, resulting in millions of dollars worth of cargo afloat in the middle the ocean.

What happens is, a whole lot of nothing. No one will actually bother to try and recover it. The cargo operators, ship owners, and owners of the actual cargo, will all file claims with their respective maritime insurance providers, and call it a day.

The same principle applies everywhere, including in cybersecurity. Past some point, trying to reduce the risk or shift the liability to somewhere else becomes more expensive than just insuring against the expected loss.

> The defense against this is to have very clear legal principles that identify the person or people fully accountable for the machine's decisions.

Be careful how hard you push for this - this is how the prosecutors in the Royal Mail fiasco drove postmasters out of business and drove a few to suicide.

That case was absolutely crazy, imagine getting accused of fraud and embezzlement just because of a computer bug.

Technology has nothing to do with this.

Before technology there was "McKinsey told me to do this". Abrogation of liability is a tale as old as time.

At least with a fully human chain of responsibility, the buck stops with someone

A computer cannot ever be made to atone for it's misdeeds

Humans can

Not really. The vast majority of adults are not at all accountable in today's society. We blame.

Someone has never encountered bureaucracy I see. Human chains of responsibility manage to dissolve blame into nothingness all the time.

Blame is not the same thing as accountability

In situations where accountability is absolutely required, you will find that people are held accountable

Often they are scapegoats, but that is a different problem

Having a scapegoat take the fall doesn't count as `accountability` in my book.

Aka the Nuremberg defense; sometimes it works, sometimes not so much.

Sarbanes Oxley in required CEO and CFO to certify company financial statements specifically to avoid the possibility of them pleading that rogue employees acted independently.

Seems like a similar legal framework could require that decision makers are held accountable for any decisions made by an AI under their control.

> Seems like a similar legal framework could require that decision makers are held accountable for any decisions made by an AI under their control.

If you delegate a decision to an AI, you are simply making the decision while trusting the AI, and should not be any less responsible than if you made the decision by any other means.

(If you are directed by a superior authority to allow an AI to make a decision but are assigned nominal responsibility anyway, though, that superior authority is, in fact, making the decision by delegating to the AI bypassing you, to anticipate the obvious “install a human scapegoat organizationally between the people actually making the decision to use the AI and the AI itself response”.)

> If you are directed by a superior authority to allow an AI to make a decision but are assigned nominal responsibility anyway, though, that superior authority is, in fact, making the decision by delegating to the AI bypassing you, to anticipate the obvious “install a human scapegoat organizationally between the people actually making the decision to use the AI and the AI itself response.

Solution is that company won’t hire you unless you are willing to take the blame and rubber stamp the AI’s decisions. Unemployed people are not in a position to protest.

Also, the point of AI is that the decisions are too complex to justify. They are grey and iffy. We don’t usually hold people accountable for anything that nebulous. Wrongly deny someone insurance coverage and they die? No consequences even without AI

Sadly at the scale of the world, we take shortcuts and need to be effective. As anyone with rare disease will tell you doctors do for ages till they get a proper diagnosis.

> Unemployed people are not in a position to protest.

Well that's easy enough, put in a safety net so access to basic sustenance can't be used as a weapon to compel people.

Unless of course, you need to compel people to work for some reason

Precisely what americans state you should not do: regulate.

There should be laws stating who has skin in the game, maybe by stating that if you take responsability for the profit by having a high salary, you also take responsability for the damage with prison.

> Precisely what americans state you should not do: regulate.

Everyone thinks the same until they're screwed over and then they want someone to do something about that. The big misunderstanding is that "regulation" is just the stuff you don't like. In reality it's everywhere the state gets involved. Every rule that the state ever put in place is regulation. Even the little ones. Even the ones that you like.

Computers cannot be held accountable more than a car, or a gun, or an automated assembly line can. That's why you have a human there no matter what, being legally accountable for everything. The human's rank and power defines how much of the risk they are allowed to or must take.

Libertarians love certain regulations - mostly the ones where the government allocates some stuff as "theirs" and uses violence to prevent other people from using it without paying them a fee.

Please go on about the regulations that libertarians love around allocation of stuff.

Sure, it's called "property" and "contracts".

But these aren’t about the government allocating stuff.

Absolutely they are.

allocate: distribute according to a plan or set apart for a special purpose

Yep, when the government makes a plan like "5 hotdogs per person per week" that's allocation, and when the government makes a plan like "each 100x100 meter plot to the highest bidder" that's allocation, and when the government makes a plan like "special rights to the creator or their nominee until 70 years after they are dead" that's allocation too.

Nice strawman, but that isn't remotely what libertarians believe.

Hell, even libertarians don't know what they believe.

Please cite the specific regulations and when they were first passed and last ammended.

> Precisely what americans state you should not do

Regulations, on a large scale, were pioneered by America as a response to Great Depression. For a long time Europe was behind the US on this front.

Regulations, actually, worked miracles for the US. But two things happened: early success that prevented further improvements (medical care), and mechanistic misapplication of the practice (over-regulating businesses like hairdressing etc.) Blinded by the later, a lot of Americans believe that regulations, in general, are bad. Well, now we see a small group of people who stands to gain a lot from deregulating many aspects of American life is about to rob blind the remaining very large group of American people :|

> [B]ureaucrats can be expected to embrace a technology that helps to create the illusion that decisions are not under their control

Isn't it the responsibility of the bureaucrat to use a computer system and whatever its output is?

"GPS said I should drive off of a cliff" doesn't seem like a very potent argument to dismiss decisional responsibility. The driver is still responsible for where the car goes.

The only case where the responsibility would shift to the computer - or rather the humans having made the computerized thingy - would be a Pentium FDIV-class bug, i.e computer system produces incorrect output+ from correct input, from which an earnest decision is then based on.

+ assuming it is indistinguishable from correct output.

> "GPS said I should drive off of a cliff"

for a bureaucrat, it's "GPS said WHO-EVER-ELSE should drive off of a cliff." Their problem.

Adding, Have a good day..

The difference is that bureaucrats make decisions on behalf of others so incentives are less aligned or not aligned at all.

If you drive a car and the GPS tells you to drive of the cliff, you wont do it because you don't want to die.

If some bureaucrat rejects somebody's health care claim leading to them dying prematurely, it's just a normal Tuesday.

Try to do something with a bank in a branch office.

Clerk: can't do anything about it, the system doesn't let me. I can get you the manager.

Branch manager: well, I can't do anything about it, "the computer says no". Let me make a call to regional office ... (10 minutes of dialing and 30 minutes of conversation later) ... The system clearly says X, and the process is such that Y cannot be done before Z clears. Even the regional office can't speed Z up.

You: explains again why this is all bullshit, and Z shouldn't even be possible to be triggered for you

Branch manager: I can put a request with the bank's IT department to look into it, but they won't read it until at least tomorrow morning, and probably won't process it earlier than in a week or so.

At this point, you either give up or send your dispute to the central office via registered mail (and, depending on its nature, might want to retain a lawyer). Most customers who didn't give up earlier, will give up here.

Was the system wrong? Probably. Like everyone, banks too have bugs in the system, on top of a steady stream of human errors. Thanks to centralized IT systems, the low-level employees are quite literally unable to help you navigate weird "corner case" scenarios. The System is big, complicated, handles everything, no one except a small number of people is allowed to touch it, and those people are mostly techies and not bank management. In this setup, anyone can individually claim they're either powerless or not in the position to make those decisions, and keep redirecting you around from one department to another until you either get tired or threaten to sue.

The real difference is that automated decisions in bank software systems are risk mitigation failsafes, and only make "negative decisions", the kind that refuse something that the bank is at a liberty to refuse. So they never need to be held accountable in this sense, they never choose the risky path.

Bank automation is like a human first making a "yes" decision, then calling a lawyer to ask if they should go ahead with it, and changing their mind if a lawyer advises not to go ahead.

    orig | auto | result
    no   | n/a  | no
    yes  | yes  | yes
    yes  | no   | no

Nothing short of a neo-luddite social movement would do the trick and even that would at best help a vanishingly small minority of the populace willing to make the kind of lifestyle changes necessary to support the ideology. Taken as a large enough group people kinda suck, they're kinda useless, and definitely lazy. It's a problem our species has been struggling with at least as long as we've had systems of writing available to document the failures of society. One could argue society is nothing more than our attempt to make people suck less.

It can also dramatically lower corruption. Can't receive a bribe from someone whose visa has expired when this person's arrest was consigned in a database.

By making someone responsible for decisions no matter the source. Perhaps someone who could be the "lead" or maybe "chief" person in charge of making decisions for the business...

The unaccountably machine is a good book on this

Good about describing the problem but the part with solutions is a bit unrealistic

> How does one counteract that self-serving incentive?

One does not, one uses a computer in decision making in order to evade accountability and profits.

See, Meta, Alphabet, the list goes on.

Funny, I'd say the above passage is the very reason we're headed towards techno-feaudalism. If you believe some mid-level bureaucrat with a computer is history's greatest oppressor, then why not have the world's richest people ransack the government? After all, they are the people's heroes, valiantly fighting oppression.

Have we watched the same events unfold?

Virtually every tech CEO was standing behind Trump at the inauguration. The takeover is the tech feudalism. They are not the heroes.

You mean like when corporations downsize and bosses / managers fire employees, we say that “immigrants / machines / overseas people etc. took their jobs”?

But when corporations grow we say it was the corporations that “created tons of jobs”?

In societies that want to promote capitalism and corporatism the everyday language we use reflects this promotion.

“I’m sorry buddy, it’s not up to me!”

https://youtu.be/wGy5SGTuAGI?t=218

"Computer says no"

https://www.youtube.com/watch?v=x0YGZPycMEU

See also https://en.wikipedia.org/wiki/Computer_says_no and the legal response to the real-world problem, the right not to be subject to a decision based solely on automated processing: GDPR Article 22 (https://eur-lex.europa.eu/eli/reg/2016/679/oj#d1e2838-1-1)

We counteract this but not letting bureaucrats make rules in the first place. Overturning Chevron was a step in this direction, but we need to go much, much further.

Overturning Chevron was specifically a giant step INTO techno-feudalism, since it means the oligarchs can only be reigned in if Congress, full of (at best generalist) career politicians, understand a technical or regulatory problem deeply enough to specifically craft legislation that literally gets it right on its own.

Chevron never was the problem.

Reply on news.ycombinator.com