Hacker News Reader

Comment by Enginerrrd | Hacker News Reader

>It's really only people where you can't tell what it does/is from the outside.

Thing is, that's not even true in that it doesn't fully acknowledge the problem. You often CAN tell information from people's outward appearance, albeit probabilistically. Therein lies the problem: you can very easily train an algorithm to be maximally right according to your cost function, but end up biased because the underlying distributions aren't the same between groups.

The issue is that as a society we've (mostly) decided that unfairly classifying someone based on correlated but non-causal characteristics is wrong, EVEN in the extreme that you're right more often than you're wrong if you make that assumption.

View on HN

>You often CAN tell information from people's outward appearance, albeit probabilistically. Therein lies the problem: you can very easily train an algorithm to be maximally right according to your cost function, but end up biased because the underlying distributions aren't the same between groups.

In fact, an AI trained on aggregate data to probabilistically infer characteristics about individuals is _literally a stereotyping machine_.

If people are upset that their stereotyping machine stereotypes people, they probably didn't fully understand what they were doing when they built it, because this is not a design flaw -- it's the design.

AI is very definitely not procedurally fair unless you can explain all behaviors. Which is why designers generally don't know what they built.

Maybe people are upset that stereotyping machines are being given harder powers to make consequential decisions about individuals.

> Maybe people are upset that stereotyping machines are being given harder powers to make consequential decisions about individuals.

Then these people should argue against AI instead of swinging the racism cudgel.

It could just be a path forward. In America a lot of the time we can't get any progress unless it's a racism issue. Things like gerrymandering or marijuana legalization get their first pushes because the most blatant group that suffers are POCs. The fact that it harms everyone is a little lost on most people, but in the end the racism cudgel can be effective for positive societal change. In this case, we can use it to get rid of automatically being identified for crimes or whatever by an AI.

I would respectfully disagree. The builder of the AI should be trained in recognizing that his discrimination machine can be used for good and for bad. If the creation shows racist tendencies, it's an outcome of the machine but a function of the (lack of) quality in the modeller. If the end result is racism, I would like to be able to point to the creator of the AI (a human) not a piece of software.

More concrete: government AI shouldn't use things like names, zipcode demographics (at least those strongly to those characteristics we think discrimination = racism) and pictures of humans in their models. Why? Because it's pretty much impossible to control your model for racist tendencies one you start there. It's in the ethics of the creator of the model to point that out and just don't do it. If you do, and all people whose name start with an M (for Mohammed) get a different category, racist is the right term IMHO.

> The builder of the AI should be trained in recognizing that his discrimination machine can be used for good and for bad.

Quite a tautology.

He's not wrong though.

It could work where a large number of AI's are constructed. A small subset of these AI's--those that can only be used for good--are used as a training set. A number of AI's that can be used for bad are added to the training set. The AI builder is exposed to this training set for a period of time, and on each exposure he is rewarded if he correctly categorizes each AI by its ability to be used for good or bad. After the AI builder demonstrates an ability to properly differentiate the AI's that can only be used for good from those that can be used for good or for bad, he is set loose on constructing a new AI, after which he is compelled to render (and publicize) a judgement on its potential use for good or bad. Alternatively, the builder can also be tasked with choosing only-good AI's from larger mixed set of good/bad AI's.

That is true but be careful with the terminology! A system trained using aggregate data to probabilistically infer characteristics isn't artificial intelligence. If the system could find causations, then there would be grounds for calling it intelligent. But finding correlations, that's just number crunching.

See also Stucchio's impossibility theorem: it is impossible for a given algorithm to be procedurally fair, representationally fair, and utilitarian at the same time:

Video: https://www.youtube.com/watch?v=Zn7oWIhFffs

Slides: https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/sl...

The general point is that you have to robustly compromise and satisfice all the goals. People tend to be rather good at it when taken as a group. (Any particular person may be bad at a given subset of all problems.)

It is a kind of optimality condition on all three goals.

The robustness additionally means that should conditions change, the algorithm usually will become better not worse and should a degradation still happen, it will be graceful and not catastrophical.

It's a hard and open problem in ML and especially ANN, design of robust solutions in the space. Most have really bad problems with it even when debiased.

Hello - do you have a good reference to this area? I bang on about similar ideas whenever allowed, but haven't found good support in the literature.

Not that I mind that too much!

Good presentation

>The issue is that as a society we've (mostly) decided that unfairly classifying someone based on correlated but non-causal characteristics is wrong, EVEN in the extreme that you're right more often than you're wrong if you make that assumption.

This is likely due to an acknowledgment of the limits of human models to account for the full context surrounding correlated-but-non-causal classifications, such that conclusions drawn from them can have unforeseen or highly detrimental ramifications.

Speaking to race in America specifically, the schema through which we judge people are highly susceptible to bias from the white supremacist bent of historical education and general discourse. This is how you end up with cycles like those within the justice system (pushed in part by sentencing software), wherein black defendents are assumed to have a higher likelihood of re-offending, therefore increasing the likelihood of any given black defendent not receiving bond or having a lengthy sentence if convicted. After all, blackness correlates with recidivism. Lying outside this correlative relationship is the likely causal relationships of longer stays in jail and lack of access to employment opportunities, which disproportionately affect black people, causing higher rates of recidivism, regardless of race.

You have other measures than prison.

You can still have enhanced vigilance without enhanced annoyance and mistakes.

There's often a superior choice lurking that nobody is thinking about, sometimes expensive, sometimes not, seemingly unrelated to such optimization. This is why ML is not intelligence, it cannot find new solutions you're not already looking for.

The main problem of the judicial and police system is it tries to be procedurally fair and still fails at it anyway.

I'd counter that in many cases it doesn't even try to be fair. It privileges the ability to craft an argument over bare facts, which immediately privileges those who can afford professional representation. At the core of the fear of a surveillance state isn't simply the loss of privacy (which in and of itself could be worth the accuracy it would bring to judicial proceedings), but the fact that it would just bolster the ability of skilled narrative-builders to pull the most advantageous facts out of context and twist them to their whims.

> we've (mostly) decided that unfairly classifying someone based on correlated but non-causal characteristics is wrong, EVEN in the extreme that you're right more often than you're wrong if you make that assumption.

Ironically, the only places where it's legally prohibited or frowned upon to use these heuristic techniques are situations that people have arbitrarily (heuristically or conveniently) decided.

For example: It's "not fair" to hire someone because they're white (and consequently have a higher chance of being wealthy and hence a higher change of being educated.)

But it's "fair" to choose a love partner based on their height, their waist-to-hip ratio, their weight (and hence having a higher chance of giving birth to healthy offspring, better physical protection, etc.).

Maybe it's hypocritical, and I don't know if that's a good thing or not. Maybe being hypocritical helps us survive.

That's not very arbitrary. If you're hiring someone, you're always in a position of power. If you're dating someone, there's no power differential (or if there is, that's a problem all by itself).

And yet from the point of view of the one who everyone is discriminating against, the feeling is pretty similar: Everyone rejects me and there's nothing I can do about it.

How are you "always in a position of power" when hiring someone? That's only true when there's more supply than demand, and it's the opposite in markets where the candidates get multiple high-quality offers to choose from.

Because you are paying that person money and have the ability to fire them. In the US you're also probably providing their health care.

I get what you're saying, but no one moves jobs every week. The sunk costs of switching employment are significant for the employee, less so for the company.

They provide you valuable work in exchange for that... Internally you are probably imagining some big corp that can pick from 100s of replaceable workers.

If you provide health care, you always have power over that person. Less so if they are relatively healthy - but any condition, theirs or a family members, means that the person has no real choice but to do enough good work to keep the job. Even if they hate the company. Even if you treat them poorly. They still must work for you. This is even more true if you have hundreds of people that will replace them and your health coverage is good enough - or at least, better than the opposition.

To a lesser degree, the same goes with vacation time and other benefits. At least in the US, anyway. This is why having some of this stuff coded into law and decoupled from employment takes some power away from employers.

Ideally there are multiple companies which you could choose from... And some might really need your specific skillset.

In practice, that's how it works, particularly at the lower end of the economic spectrum. If that wasn't the case then the concept of a minimum wage wouldn't be necessary - the market would take care of it.

Of course, this is a point of view and not everyone agrees with me, but to me it appears that for a chunk of the population the available jobs do not pay well enough to meet a certain standard of living.

> it appears that for a chunk of the population the available jobs do not pay well enough to meet a certain standard of living.

That is definitely true, but it's also pretty much the exact opposite of "always".

> If you're hiring someone, you're always in a position of power.

What? Can you explain the reasoning behind this statement?

If the candidate had more power, she would set up interviews and force the employers to impress her.

Of course, that scenario happens only in extreme edge cases. Even in a booming economy with a shortage of workers, John Doe is not going to be pursued aggressively to fill the Senior Marketing Manager role.

This makes intuitive sense: employers have a ton of money, so people come to them.

Right now I have a client who needs to hire truck drivers and can't do it fast enough. I asked him what he'd done to make his company the most attractive (pay, technology, perks, etc.). He said he's done nothing.

> John Doe is not going to be pursued aggressively to fill the Senior Marketing Manager role.

That's exactly what recruiters and headhunters do. Aggressively nails it.

Recruiters and headhunters don't necessarily seek people to fill a role. They seek people to fill their batch of application forms to send to those actually doing the hiring.

Recruiters don't replace the interview process, where the dynamic is that the applicant is the interviewee. They only change the way the applicant discovers the job.

Most people will never be headhunted.

By that same logic, it's unfair for attractive people to choose who they date, because they're in a position of power.

If candidate had more power, companies would create whole section of company to try to find and hire talents. Companies would literally pay to find candidates.

There are two possible situations:

A candidate has multiple job offers, and decides which one to take.

The company interviews multiple people for a single position, rejecting the others.

There are only a small number of sectors where the first situation is reasonably possible, a lot of us on here are incredibly fortunate that engineering happens to be one of them. For the majority of the job market (by volume of people rather than volume of money), it takes people attempt after attempt to get a job. They don't get to choose between multiple offers, they have to take the first thing that will allow them to pay the rent, and then they have to hold on to it.

This is over stated:

"are situations that people have arbitrarily (heuristically or conveniently) decided"

The Holocaust was not convenient, even to those who were for it. Slavery was convenient for those who benefited from it, but not those who suffered under it. Over the last 200 years racial categories lead to many millions of dead. This is not merely a matter of convenience.

The holocaust and slavery fall in the realm of physical violence, or at least coercion.

Violence and coercion don't logically follow from racial differentiation. One may point out the differences between populations of different races, but that wouldn't justify attacking any individual from those populations.

My comment is framed inside that basic (and obvious) principle. Choosing a partner or an employee is not a violent or coercive act.

The hiring vs mating issue is perhaps simpler than you picture. US federal discrimination law only applies to companies with more than 15 people. If we regularly married 15+ people at a time, we might very well put legal restrictions on your mating choices. The more personal the decision, the more agency you get.

wow, interesting. Why does it only apply to companies with more than 15 people? Is it the idea that you're more likely to have family help (and only family being willing to help) when your company (more small business than traditional startup) is this small?

If you are starting a small company you either pick people you already know (whoever those might be) or maybe a few random experts with very specific skillsets. There is no place for you to actively discriminate someone that would have maybe been a better pick, just because you didn't like their skin color or gender... Or if you still do it comes at your own loss.

> if you still [discriminate] it comes at your own loss.

This applies to companies of any size.

A job contract is naturally a relationship between two entities: employee and employer.

Sure, some people in some countries have tried (sometimes successfully) to undermine this principle, but it's akin to forcing people to marry in groups.

While (modern? ideal?) marriage is a peer-to-peer relationship, the relationship between employee and employer is unbalanced, more like pet-owner.

Societal restrictions on contracts are a way to balance this. Virtually all societies have some form of this, outside utopian ultra-liberal hellholes.

Turning it around though. It's entirely fair to hire someone because they are well educated, which presumably means you're disproportionately hiring white people.

Europe has the concept of indirect discrimination, which could make that illegal, certainly things less central to the role could amount to indirect discrimination.

> The issue is that as a society we've (mostly) decided that unfairly classifying someone based on correlated but non-causal characteristics is wrong, EVEN in the extreme that you're right more often than you're wrong if you make that assumption.

That sounds like the wrong basis for calling it extreme. It's not at all extreme to say that classifying an interview candidate based on correlated but non-causal characteristics is wrong, regardless of the statistical significance of those correlations.

I just mean that I'd guess most stereotypes have a correlation nowhere near 0.50, but it'd be wrong to use them to screen candidates even if it was 0.50 or greater. I used the word "extreme" to convey the sense that such a scenario of stereotype accuracy is very unlikely.

Most stereotypes are lagging indicators that do not tell us about the future, as the history of Fenty Beauty makes clear.

I don't believe Fenty Beauty has been discussed on Hacker News before seeing as how it's a line of cosmetics, so maybe you could expand on what you know about its history that others may not have been following in as much detail?

In 2017, Fenty Beauty found an underserved market (dark-skinned women and high-quality makeup) and made a killing selling them what they wanted. If you relied solely on history and stereotypes, you would believe nobody could make $500 million dollars that way.

https://www.fool.com/investing/2019/05/15/rihannas-fenty-is-...

Thank you very much!

> EVEN in the extreme that you're right more often than you're wrong if you make that assumption.

And this is actually the rational thing to do. Reason: There are two potential errors involved here: labeling an innocent person a criminal (false positive) and labeling a criminal as innocent (false negative). The key is to realize that the cost of these two errors is not the same. For instance, treating an innocent person as a criminal could be much more expensive for the person and society than not detecting a criminal with a given classifier. For that very reason, we have the presumption of innocence as a principle in law. As a consequence, you don't want to select a classifier based on just the rate of errors overall, but you want to incorporate some kind of loss function that minimizes the cost for individuals and society. Under that loss function, the best classifier may actually be wrong more often than some other classifier.

I'd just flag that "EVEN in the extreme that you're right more often than you're wrong" is not a meaningful line (if someone wanted to apply it) because the fact is that it's not the number of wrong calls or right calls that matter here - it's which ones and when. For example the project under discussion classifies a photo of my mother as a young woman as a "flibbertigibbet", which is amusing, as a way to tease her now (she is a 70 year old ex prison and probation officer and needs no protection from teasing).

However, had this been done to my daughter and the same result obtained (in fact the result was "blue" but there you go) by someone assessing her potential as a candidate for university - well there's damage.

The fact is that we live in a society

In many cases, being logically correct is more important than being politically correct.

For example: It may be politically correct to drive down a bad neighborhood in the middle of the night, but it isn't logically correct.

True, but poorly stated. Which is to say, you aren't wrong, but you're missing the indications of nuance which are really, really important to the personal liberty of not being defined by superficial traits.

It logically correct to do that when it's your neighborhood and you're driving home.

But note the implicit bias in your own comment: you assume yourself and all the readers of the comment are not people who live in bad neighborhoods.

>But note the implicit bias in your own comment: you assume yourself and all the readers of the comment are not people who live in bad neighborhoods.

that assumption results simply from the need for 'bad neighborhood' to be a negative scoring action.

if you oversimplify it to get rid of 'implicit bias' (which I don't agree exists in the example), the results turn into near-meaningless babble.

"For example: It may be politically correct to do something that ignores statistical dangers in favor of the promise of human goodness, which may result in the possibility for more personal endangerment than other choices, but it isn't logically sound to ignore such statistics for the hope of a less biased personal experience."

The example requires the person driving to be detached from the bad neighborhood that they have a choice to drive through. How that isn't an obvious requirement for the example to have merit is beyond me.

While there may be some bias, it is as much yours. Even people who live in bad neighborhoods avoid driving through them when possible.

That is not technically even true. If you live in a bad neighborhood it's still probabilistically better to drive home thru a good neighborhood than thru another bad neighborhood.

Did you live in really bad neighborhood? I did. Out of two bus stops I always used one that required to cross two streets with no crosswalks. The other required to walk right through the middle of my lovely vicinity with 50% probability of giving something up to ‘charity’.

Well, it's especially in those situations that you need institutional controls to forbid the logically correct choice.

After all, you don't really need a law telling companies they aren't allowed to hire infants as senior officers - it's already not in the company's interest to do so.

However, when there is a logically correct but politically incorrect decision that the company could make, it is now that you need laws to prevent the company from taking that choice.

Of course, as the weight of an institution's decisions goes down, so does the need to police its actions. In particular, it is rarely necessary to prevent an individual person from acting on their biases.

Applying this to your example, if we had an AI that should suggest your best route home, and it avoided a short route through a bad neighborhood, that is likely ok. However, if a municipality used the same AI to decide where to prioritize changing street lights, that should be prohibited.

> The issue is that as a society we've (mostly) decided that unfairly classifying someone based on correlated but non-causal characteristics is wrong, EVEN in the extreme that you're right more often than you're wrong if you make that assumption.

Sorry, but it's not a decision. Science has found repeatedly that using outward characteristics does not work as a good classification measure. Society simply enforces not making bad judgements. See:

Phrenology, Racism -- "appearance implies certain things about someone's character and intelligence" (https://en.wikipedia.org/wiki/Phrenology)

Sex -- "A person's gender presentation determines their gender or sex" (https://en.wikipedia.org/wiki/Transgender_history), "A person's genitalia determines their chromosones", "A person's chromosones determine their sex and their ability to give birth" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2190741/).

Wealth -- "That person is wearing bad clothing/living an unassuming life therefore they must be poor" (https://www.npr.org/2018/12/29/680883772/social-worker-led-f...)

Intelligence -- "This person scores badly on intelligence tests therefore they must not be worthwhile" (https://en.wikipedia.org/wiki/Richard_Feynman, https://www.psychologytoday.com/gb/blog/sudden-genius/201101...), "This person appears to be unskilled or incapable of many tasks therefore they must be totally unskilled" (http://en.wikipedia.org/wiki/Savant_syndrome)

> Science has found repeatedly that using outward characteristics does not work as a good classification measure.

Science has found that, on average, it works [1]. Of course there will be many cases where it fails - that's how statistical inference works. Whether that makes it a 'good' measure, by whatever standard, is a different question. But there is no doubt information can be inferred from appearance.

[1] http://www.spsp.org/news-center/blog/stereotype-accuracy-res...

>Sorry, but it's not a decision. Science has found repeatedly that using outward characteristics does not work as a good classification measure.

Lol, what? Generically, that statement is almost certainly false more often than true in general in science. But I believe you are restricting yourself to social psychology?

You then presented a long list of examples taken from the tails of certain distributions to refute an argument that said distribution exists and has an average? I didn't even name any particular distributions. You're thinking appears flawed and emotionally driven, and most unfortunately, that's the type of thinking that will lead you to building biased systems.

Here's the point you missed the first time around: There are going to be outwardly visible characteristics that ARE correlated with some factor of interest, to the extent that training a machine learning algorithm based on a cost function that uses predictive accuracy alone WILL result in a system that assigns what society would consider an inappropriate importance placed on non-causal but correlated parameters.

Here's a real world example that might help you understand why this is important: (Data taken from: https://en.wikipedia.org/wiki/Incarceration_in_the_United_St...) Because blacks are over-represented in the US criminal justice system (40% of the prison population vs 13% of the population) and because part of what defines "black" is the outward appearance of certain facial features, a facial-recognition algorithm which is trained to recognize criminals, with a cost function based on prediction accuracy alone, and facial features as input parameters is going to have false positives that over-represent blacks. Does that sound like something you want? Because denying the underlying distributions is going to lead to exactly that.

It's very important to consider this when you develop a training set, for fucks sake. It might work something like this: Take 100 innocent people's faces at random. (On average it will have only 13 blacks) Then take 100 random criminal faces from inmates. (On average it will have 40 blacks.)

Then mix up the groups into your training set and assign a prediction score 1 or 0 depending on whether or not your classifier has correctly predicted whether or not a face was in the criminal group. Then, based on no other feature than race, your neural net can get better performance based solely on guessing more often that black people are criminals. That's not a good thing.

Do you get it now? The likelihood of being falsely identified as a criminal is greater based on the non-causal but correlated variable of being black. And this has happened several times already! You can't keep pushing this narrative that neglects the underlying statistics because of your beliefs, or people will keep making racist systems.

It seems that some tasks are just inherently racist (and sexist, etc.), and we should be able to identify them before someone inflicts it on society.

Trying to identify potential for criminality based on looks may well work, identifying and using the underlying racial biases. And if those systems are used, where people identified by these systems are more likely to be identified as criminals and investigated, we end up with a feedback loop and, over time, the racial biases in a society that uses the systems will get worse. More black criminals will be caught, as they are more likely to be suspected as criminal, making the racial biases worse for blacks. While the opposite happening for white faces.

Similar social evolution would happen if you pre-screened job candidates, over time magnifying existing gender and racial biases. I've seen in some cultures it is required to add photographs to job applications, but I think it a good thing that this practice is discouraged in western countries.

This is truly where the danger lies. You can say that you're trying to make rulesets that are accurate to the world we live in, without acknowledging that you fall heir to, and may in fact be producing this generation's version of, historical policy decisions that contributed to or outright created racial division and disparities in the first place. Then and now exist an appeal to empiricism that takes the current state as the natural order, and not one manufactured by past decisions which aimed specifically at a particular, and not altogether organic, conclusion.

So, I disagree with the "inherently racist" portion of your argument. You can evaluate the classifiers in a race independent manner, i.e. take a look at the metrics stratified across race. I'm going to proceed here assuming that the classification task is possible.

Say in your example of criminality, if black people were more predicted by the classifier, it isn't racist if it accurately reflects the base distribution. There's a line to be crossed here, and to me it's if that application starts to significantly (where the boundary lies here is up for debate), and directly affect the non-criminal portion. I'd say that if the misclassification error is similar across racial lines, then there is no issue.

Additionally, I don't quite agree with the "making racial biases worse" argument either. The way I see it, we already use racial heuristics in law enforcement. With automated, replicable tasks, we can at least quantify the degree of bias and correct for such.

The main question remains: what price do you want to pay for procedural fairness, why is it even a major goal?

Most people would probably opt for utility mixed with representational fairness of some degree even when it means law applies to some degree differently to groups and special cases.

Justice is not fairness. It has an institution called motive. (Which is often overly simplified or ignored.)

It incorporates elements of fairness. And it is hard to train. Not everyone can be Solomon, for example.

I haven't seen a single use case, or a proposition, to use AI classification on such a raw task as classifying whether someone is a criminal or not just by the photo. Any sane person would see that such an endeavor would by itself present so many problems in a society, way longer before we even get to the racist bias of a statistical distribution. So what is the danger really?

Proposition? Does this count as a proposition?:

https://www.newscientist.com/article/2114900-concerns-as-fac...

Ok yes well that one is absolutely crazy and scary. Regardless of whether there are race biases in the database.

Yeah I didn't pull that out of a hat. The specifics were arbitrary, but I chose that as a real example of systems that people already tried building. THAT'S why understanding Bayesian statistics and underlying distributions is super important to consider when you might inadvertently create biased systems.

That is less relevant than the goal, which is to have a better society while not violating human rights, for a very wide definition of human.

In short, being humane. If a certain degree of racism or stereotyping is necessary for that goal (definitely not 100%, but something low), then so be it.

Currently multiple social system in the USA are considered much too racist.

You're saying correlation is not causation, a very well known truism. Surely you can't be seriously saying that society or anything else effectively prevents people from equating the two?

You have such poor use of sources to underline your claims it is almost incredible.

What underlines it is seemingly a lack of understanding of statistics, becausee the most "examples", if they can even be called that are from the realm of non-common outlier cases. That's not what statistics operates on mostly - it operates on common cases that cover the largest part of a distribution. Not outlier cases on the ends of a distribution.

In every case you gave an example of some uncommon exception, as if that would somehow be an argument for the rule existing?

Reply on news.ycombinator.com