Comment by carapace | Hacker News Reader

Comment by carapace | Hacker News Reader

Parent Full thread

"The target of the Jihad was a machine-attitude as much as the machines," Leto said. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgements. Naturally, the machines were destroyed."

https://en.wikipedia.org/wiki/Butlerian_Jihad

Seriously though, how is this not the crudest sort of cargo-cult machine worship? I'm not paying taxes to be judged by a goddamned machine. It's crazy. If the people we're paying to administer the system are so degenerate that they would delegate their most essential function to a machine then we need to vote them out in favor of sober, responsible, educated adults. It's like they're admitting that they can be replaced by a script!!!

Knowing what I know about computers, and computer modelling, I'm appalled that people would make, buy, and employ these sorts of "voo-doo" machines.

(Thin sans-serif body font means you hate your readers, and making it grey means you really hate them.)

> I'm not paying taxes to be judged by a goddamned machine

I'd be shocked to learn that the IRS does not use some form of mechanical computation to determine probabilistically whether or not a given taxpayer should be audited for possible tax fraud in a given year.

Maybe so, I'm not privy to their methods, but I wouldn't be surprised that they use, e.g. statistical models, computed on machines, to help them figure out whether or not to look more closely at a given taxpayer.

But however unpleasant it may be, an audit is not a punishment. (Someone is abusing their power if so.) Not anymore than being selected for jury duty is a punishment.

Also, you used the word "probabilistically". We should not be assigning sentences based on the roll of a die, eh?

Insurance companies use machines and statistical models to set rates. But they also have strong feedback to improve accuracy. They lose money if they get it wrong.

Here we are talking about judgement.

I know far too much about computers to ever credit the idea that these machines can replace a human judge. The people who have put these things into practice are deeply incompetent and should be voted out of office and replaced by folks who won't devolve to crude machine-worship.

> Also, you used the word "probabilistically". We should not be assigning sentences based on the roll of a die, eh?

Probability is not "roll of a die", it is how reasoning actually works. Human judgement is nothing but a heuristic-driven, semi-conscious probability calculation. When you have data that's more quantified than "vague gut feeling", doing things the right, formal, explicit way leads to better results - that's why machines are getting much better at reasoning than humans already.

So if IRS wasn't using probabilistic models to decide whom to audit, I'd say they'd be sucking at their jobs.

With respect and affection "Human judgement is nothing but a heuristic-driven, semi-conscious probability calculation." is a statement of faith, not an established fact. (How would you establish it?)

Machines don't reason, they compute. It's not the same thing at all. We can reason about things that cannot be represented in any symbolic fashion. Put another way, no machine can tell you: which schools to attend, whether to marry that person, what to have for lunch, the optimal route to work.

Deciding whom to audit is different than deciding how long you should stay in jail.

An audit may not be a punishment, technically speaking, but neither is being predictively policed, technically. Neither is being stopped for a brief patdown because a suspect in an armed robbery is black and you happen to be black and within the vicinity.

The latter situation is most definitely an algorithm whether it be administered by a computer or human -- e.g. 1. is there a person in front of me. 2. Is this person male? 3. Is this person dark-skinned? 4. Is this person within X meters and inside a timeframe of Y minutes from a reported crime?

As for making judgments probabilistically; hate to break it to you, but that's already the status quo. Exhibit A: Probable cause.

And probabilistic thinking is most definitely behind sentencing guidelines. Let's take the recent and still controversial case of Brock Turner, the ex-Stanford swimmer convicted of sexual assault [0].

Aren't the following aspects of the Turner case all a manifestation of probabilities?

0. That his bail was set at $150,000, based on the likelihood of an accused (and not yet guilty) rapist of his status would break bond.

0.5 The likelihood that he did indeed sexually assault the victim and was not ignorant as to her state of consciousness or consent.

1. The recommended minimum sentence of 2 years in prison for his convicted crime.

2. The probation officer's recommendation that he be given a "moderate" county jail sentence.

3. The judge's determination that Turner, based on his lack of prior criminal record, be given 6 months in county jail.

4. The county jail's determination that Turner serve only 3 months in county jail because of good behavior.

5. The post-incarceration penalty that comes with Turner to be thought of as a Tier III sex offender, which means he has to register 4 times a year (presumably based on some statistic that that's enough polling frequency to adequately track him), for the rest of his life.

Note: I'm not implying that you agree with any of these aspects of the case. Just saying that if you're concerned about statistics and likelihood playing a part in our criminal system, then that ship has long sailed. In defense of the machines, if a machine had been the sole determiner of Turner's sentence, we'd at least be able to audit it and know how it weighted things. With the human judge, we don't have a clue as to whether the judge's personal relationship with Stanford was a factor, or even if he read the probation officer's report correctly when giving the seemingly light sentence. Hell, we don't even know how the probation officer interpreted the victim's wishes [1]

[0] http://www.daytondailynews.com/news/news/crime-law/greene-co...

[1] http://www.cbsnews.com/news/probation-official-called-brock-...

I don't think that anyone is proposing a transparent algorithm that can be audited. They're all proposing trendy, proprietary machine learning techniques sold as AI. If you're just talking about boring transparent algorithms, we've always had the technology to devise them - it's called pen and paper. Mandatory Minimum sentences and Three Strikes laws are clearly punishment algorithms.

The point you're making about the Brock Turner case is well taken; the judge was another standout Stanford athlete in a sport that doesn't have a professional level. My guess is that these algorithms won't be devised by poor black people who dropped out of high school, but they will have a lot to say about how much worse poor black people who dropped out of high school are than people from the demographics of the people who do devise them.

It's also important to note that the algorithms today cannot be audited seriously. We know, for example, that many municipalities have been caught installing red light cameras and then adjusting the yellow light timing to maximize revenue rather than maximize commuter safety. We know that some laws, particularly drug laws, have consequences designed to punish "undesirable groups" (minorities) rather than make streets safer or protect the lives of citizens. So having pseudo-AI that is also a black box isn't necessarily worse than what we have today. Because as of today we can't really audit the algorithm itself, only its output.

Dude you are working way too hard to do... what?

What's your angle?

For me, it's really simple: The day that a machine decides how long I spend in a cage we are going to have a serious problem.

Sorry, you said earlier that you were someone who understood computers well enough to know that you would never want one to replace human judgment in matters of justice. I thought you'd be interested in knowing that computation (delegated to machines for efficiency and consistency) is actually part of the justice system. The Bureau of Prisons has a department that is named, "Designation and Sentence Computation Center".

Maybe they don't actually use physical computers to perform the computation (though that'd be a bit weird if they didn't), but I think that's an irrelevant implementation detail as the Supreme Court has upheld the use of an unyielding mathematical formula to determine the length of a prisoner's sentence:

https://www.supremecourt.gov/opinions/09pdf/09-5201.pdf

https://www.law.cornell.edu/supct/html/09-5201.ZD.html

Hmm, well, IMHO that's all bad too. Not sure what, if anything, to do about it.

This doesn't seem as bad as the OP, but it's still an abdication of responsibility. Automated authority.

Let me point out something: why are these systems being deployed to decide sentencing, as opposed to trying to figure out what will actually affect recidivism?

I haven't read about that in any of the articles on this: is anyone paying attention to e.g. whether a given sentencing policy is actually reducing crime?

I don't doubt that machines can help us understand and manage ourselves and our complex society, but this seems like machine-worship to reinforce our atavistic ideas around crime and punishment.

He is trying to think rationally. Sometimes attempting to find the right answer, rather than simply the one that feels good, is in fact "hard".

I'm glad some people put the effort in rather than just chasing positive feels.

Mr. Stucchio:

Mass democracy is the root of this problem. One proof is observing the way in which democratic decision-making has devolved chiefly into emotional appeals to in-group moralistic bromides. To me, the degree to which one seriously desires rational thinking as a basis for lawmaking is the degree to which one desires a reduction in the number of cooks in the kitchen.

Rationally speaking it would be irrational to ignore emotional responses when reasoning about human behavior.

Rationality can lead you astray, you have to listen to your heart.

Emotions can lead you astray, you have to listen to your head.

>But however unpleasant it may be, an audit is not a punishment.

If every audit was accompanied by a check for $3000 or so, a large portion of the people audited would feel neutral about the process.

If they did, I'd be shocked if large tax cheats weren't tailoring their cheating specifically to the algorithm, and the only people that it caught were small timers and people who made honest mistakes.

This is already the case. Humans doing judgement calls are an "algorithm" too. Moreover, there are only two algorithms you can't game - one that perfectly addresses and solves explicit and implicit aspects of the problem (impossible in practice), and one that is perfectly random.

Computer algorithms are much easier to improve than low-paid human clerks, so there's that too.

> Moreover, there are only two algorithms you can't game - one that perfectly addresses and solves explicit and implicit aspects of the problem (impossible in practice), and one that is perfectly random.

Care to explain?

The first one is ungameable by definition. Also practically impossible to make by the very same definition - for one, we always work with imperfect information, but moreover, humans are damn good at redefining the rules of the game on the fly.

The second one follows the principle of American Army - "If we don't know what we are doing, the enemy certainly can't anticipate our future actions". A purely random selection cannot be gamed directly because it doesn't change its behavior in response to external input. However, it certainly can be meta-gamed; e.g. some people could take advantage from the very fact such algorithm is used in the first place. This circles back to the same point I made about the "perfect" algorithm - humans are just too damn good at metagaming.

Still - my primary point was that a) humans are a gameable algorithm too, and b) humans are pretty low-hanging fruit as algorithms go. We can, and should, do better.

[0] - http://www.allproudamericans.com/paimages/here-is-why-americ...

I completely agree that those are not gameable. The thing I wondered about was why there can't be any other ungameable algorithm.

I believe the two cases are provided (perfect algorithm and random algorithm) exhaust the list of ungameable ones (though they're still meta-gameable).

I would be. I would find it much more likely that the IRS uses some sort of much cruder algorithm in tax auditing, which is what is intended to detect fraud and appears to be non-functional at this point.

> It's like they're admitting that they can be replaced by a script!!!

Well yes, evidence suggests that in many cases they can be.

Reply on news.ycombinator.com