Hacker News Reader

Comment by andrewflnr | Hacker News Reader

I think Yann and his critics are focusing on different questions. IIUC Yann is looking at the narrow engineering perspective of how to build an fair ML system. Gebru et al seem to be talking about a much more abstract, societal trend-oriented, "let's think about if we should not just if we can" type of question, though it's still pretty fuzzy to me. Anyway, the epistemological standards of those questions are basically incompatible (incommensurable?), so discussion is impossible unless they agree on which one they're trying to answer. I'm starting to notice this kind of thing a lot.

View on HN

I think there's two different issues here.

One, if datasets are biased --- if you build your system to only work on white males --- then it may have suboptimal results for other groups. This is a common problem: you use your company's faces, or college students enrolling in data gathering exercises, etc, who are not representative of the population at large. We can fix this by being careful about dataset bias.

But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets? If you train something to predict, say, odds to default on a loan-- will it figure out things that correlate to race and be making mostly racial decisions? Different races do have different default rates, but we've decided as a society that it is unfair to use race to determine an individual's probability of default. But if we choose things that are correlates of both default rates and race, when is that fair and when is it just veiled racism (redlining)? What things are measuring a causal relationship and what things are just racism in disguise?

This second problem is much worse with ML, because we have the ability to accept a whole bunch more things into our models and explaining the rationale of why decision are made is much harder.

And of course, the first problem-- both bias in datasets during use and biases during research and development -- makes the second problem worse. It can't even necessarily be addressed by broadening the dataset and retraining: if, in this case, you do your research and training with just white faces, and report positive results, it may not generalize to work as well for everyone with a broader dataset.

Is that attenuated by making the system "blind" to race in those decision-making processes?

It's pretty hard to make the system "blind" to race. If you blind a system predicting recidivism to race, but it is able to see a lot of things that are correlates of race... it can end up being racist anyways.

E.g. for a non-ML example -- redlining is theoretically "blind" to race, but makes extremely racist decisions.

That is difficult to do because there are many things that correlate with race. So even if you remove race explicitly, things like neighborhoods, level of education and such end up being proxies. Removing race from the model is a very hard problem.

One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't.

I guess the question is why would that be germane to issuing credit? Is it correlated to something that indicated actual credit risk? If it makes erroneous assumptions correlations, would it not behoove them to eliminate the contamination?

Sure: some of your purchase history definitely indicates behaviors that indicate you're a credit risk. Some of your purchase history indicates you're likely black (and if you're black, your credit risk is, empirically, a little higher-- even if it is not fair to not issue credit to people because they're black).

ML will figure out both but can't explain the hidden variables: it'll figure out behaviors that directly indicate higher risk, and behaviors that indicate you're a minority that tends to be a higher risk.

Is it too entangled to say disregard anything that leads to a path that makes inferences based on demographics? Although I suppose age is also tough to avoid in assigning risk.

Almost anything you'd use is correlated to demographics somehow or another. We tend to think certain systems are "OK" and not too racist, because they have a well-articulated rationale independent of demographics. But ML is not very good at explaining rationale for decisions, so we don't have the benefit of that argument and it's difficult to even empirically understand how biased the system is.

This is a hard problem.

Basing your entire decision on something as simple as the applicant's income will presumably produce outcomes that differ with race. Is that what you mean by racist? Or do you mean the system identifies some individuals as high credit risk when they aren't and may actually lose money versus a more colorblind algorithm?

What's racist is squishy.

Many people would say that identifying people who spend a lot of money at bars or casinos as a credit risk isn't racist, even if it happens to pick up more minorities. The mechanism for the credit risk and behavior seems tightly correlated.

Many people would also say that identifying people who spend some money at clothing retailers that market to minorities as a credit risk is racist. Here, the relationship seems like a hidden way to spot minorities, who happen to be more of a credit risk.

When banks drew red lines around all the minority neighborhoods and didn't lend to people there, because they couldn't consider race anymore, most thought this behavior unacceptable, even though it did genuinely reduce credit risk for banks. ML can't explain its rationale, and very often does the exact same thing inside an opaque box.

Presumably black people who are great loan customers would be incorrectly refused by a system based on fashion or addresses alone, so your answer seems to be that yes, racist methods actually lose the bank money.

My tendency is to think that since it is optimized to maximize profit, the more features about a person we get in the data, the less "racist" the system becomes under this definition. Yes it's more possible to "hide" racism in a complex method using tons of features, but increasingly less likely it would happen. If we can use income and debt and whatever other things to make either a good classifier that ignores race, or an inferior one that sneakily identifies race and bases the decision on that, mathematical optimization of the model should result in the former. ML can be trusted more than humans in this situation, not less.

Of course there's still the issue of the data being biased, which is where it all started.

I wonder what a court would rule if you went through your algorithm line by line and demonstrated that is was only designed to maximize profit, and the fact that it was racist was only incidental.

I'm guessing it's illegal to discriminate based on race, gender or ability no matter how much it extra it costs a business, whether its deliberate or not.

Well in places like NYC the laws are strict regarding rental housing. Landlords need to establish consistent requirements then accept the first applicant that fits them. Typically they require the income to rent ratio is greater than three or something like that, which undoubtedly has a disparate effect when it comes to people of different races given how high the rents are and how incomes differ. Are you calling this incidental racism?

The issue is that "fashion for minorities" is correlated with being a minority which is correlated with poverty which is correlated with high credit risk.

This makes "buys fashion for minorities" quite an accurate predictor of high credit risk, but also a profoundly racist one.

Yes I got that. The question is whether simply discarding the overtly-race-based features from the data will be enough.

> Presumably black people who are great loan customers would be incorrectly refused by a system based on fashion or addresses alone,

Strawman. The question is whether predictions are improved by using race (or direct proxies for race), not whether it'd be wise to use proxies for race alone.

Loan default rates are higher for disadvantaged minorities, even after controlling for many, many other variables (income, neighborhood, level of education, etc). Therefore, using race (or inferring race) improves prediction quality, but is ethically dubious.

> Strawman.

Can we skip this kind of snippy arguing please? Anyway I take it you agree the statement is true but don't want to.

So now you're saying those great minority loan customers are impossible to identify? I think you just need to figure out what information is still missing. What's the effect size at this point anyway?

> > Strawman.

> Can we skip this kind of snippy arguing please?

If you don't start off by willfully mischaracterizing your opponent's argument in order to be able to more easily refute it, I think you'll find that they accuse you of this less.

I think you're just willfully missing the point.

Ideally, you'd just lend money to the people who will pay you back. Unfortunately, we can't predict this perfectly. Adding race, or proxies for race, to the things you consider improves your predictions somewhat.

Nope it was a completely honest question and what I considered a constructive line of thought. You should assume good faith. I asked about two competing definitions of what is racist and you seemed to prefer one over the other.

And what have I claimed to "easily refute" exactly? More like I ran with the definition, and considered how to address the problem as stated. I said more features were needed, I didn't say enough features were currently used. You keep pointing to a dichotomy between perfect and flawed, while I was talking about relative improvements. There isn't even necessarily a disagreement there.

I sez, "other variables + race makes better predictions of default risk than other variables alone". U sez, "LOL -only- using race will cost banks money" I sez, "???"

By the way many methods are excellent at explaining their decisions. The favorite is decision trees which break the decision up into discrete decisions for individual features. E.g. a person was put in class 1 because their income was above X, their savings was above Y, etc. Other "shallow" methods like regression are fairly easy to interpret as well.

It is mainly deep learning for which this is difficult. Which I'm sure is the method in the twitter discussion, but that is a different kind of problem where interpretability doesn't have a clear use.

When we say "ML", we usually don't mean regression. We not mean deep learning, but at least we mean SVM, SGD, etc.

They may be less inscrutable than deep learning, but can easily still draw a box around most of the black people in a clever, non-obvious way.

Do you have a source?

e.g.

https://consumerist.com/2008/12/22/amex-lowers-your-credit-l...

Capital One has admitted doing the same, as well as a number of other lenders.

There's been analyses done since, that show that when controlled for all other variables in credit reports, credit issuers issue less credit to blacks.. e.g. Cohen-Cole, 2011, Credit Card Redlining, Review of Economics and Statistics

Credit Card applications do not ask for race. I suspect it’s more likely based on zip code or similar.

I said "One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't"

You asked for a source.

I provided a source of the same. Yes, things like purchase histories are used as proxies for race and used to deny credit to mostly people of color.

Then you go here:

> Credit Card applications do not ask for race.

which was never asserted.

> I suspect it’s more likely based on zip code or similar.

Yes, zip code can be another proxy for race.

Yes, thanks for the example of the former.

As for the latter, no, it’s not a proxy for race. It might be highly correlated with race, but that is very different. I don’t know why everyone does these gymnastics to find a racist angle to everything, but it’s nonsense and it’s very harmful to spread that narrative.

Nobody is searching for black zipcodes and using that as an input to deny credit. The goal is to increase revenue by giving as much credit as possible with the least risk possible, so it wouldn’t even make sense.

Scoring based on where you shop is still behavioral analysis. I think it’s a bad policy, but it isn’t targeting people of color. Your zip code affects insurance rates too, and that’s based on claims. It doesn’t cost more to insure your car in south side chicago than beverly hills because there are black people there, it costs more because there are more claims. The same effect is seen in “white” zip codes that are in areas with severe winters.

> I don’t know why everyone does these gymnastics to find a racist angle to everything, but it’s nonsense and it’s very harmful to spread that narrative.

Black people default on loans more, adjusted for income, education status, etc. But we've decided it's socially un-okay to ask people if they're black and adjust how much we're willing to loan based on the answer.

But instead, we measure all kinds of ancillary things, many of which are highly correlated with being black and not obviously correlated to ability to repay, and dump them into a model, where we come up with weights that make basically the same decisions as if we'd asked if they were black. Is this is fundamentally better somehow?

Mind you, I don't have a wonderful answer as to what to do: it's important to accurately price credit risk. But it's also important for society to treat minorities equitably, especially when inequitable treatment might reinforce the very problems/reasons why some minorities are worse credit risks.

It might be better to explicitly have the "is African American?" question in the model, because then regulators, policymakers, the public could perhaps eventually know the exact contribution of this factor... while approaching it obliquely makes the effect far less clear.

Or is race the proxy?

It's really, really hard to erase biases that are deeply systemic.

For example, from the outset, would you object to an AI that made decisions on how harshly to sentence someone based on age & number of prior crimes?

What about hiring or admitting people to college based on the results of an IQ test?

Yeah, those end up being badly biased, this first has been studied and the second is the reason for dropping SAT/ACT scores--every mental ability test correlates with IQ.

The more interesting thing, IMHO, is that the opposite is also helpful. For example, if you help all poor people equally, you help even the playing field by disproportionately helping out all disadvantaged groups.

Application of law should always have room for judicial discretion. Mandatory minimum sentences should not exist, but then you get outrage over some light sentence here people get all riled over about and want to impeach judges and all.

Education is different. IQ shouldn’t be relevant but competency and aptitude should and in addition we should recognize some people are better off going to vocational school where they may do better economically during their lifetimes.

>For example, from the outset, would you object to an AI that made decisions on how harshly to sentence someone based on age & number of prior crimes?

Yes. This is an overly simplistic system for which it would be hard to justify the cost of developing.

>What about hiring or admitting people to college based on the results of an IQ test?

Yes. For the same reason.

The first one was intentionally minimalist in order to see how little it took for racial bias to be reconstructed.

For the second, any mental ability test correlates with IQ, so you end up with a different set of difficulties. There have been attempts to, e.g. correct for cultural bias in the tests, but these actually made the problems worse.

I'm not presently aware of anything that makes that situation better.

Assuming good faith, I really think there is a gap in understanding why an "objective" viewpoint can be seen as blinkered or prejudiced.

If you're looking at things that correlate, you've got a lot of choices. Saying that your data set is representative, your algorithms are correct, doesn't mean your choices of what correlations to use are objectively or morally correct.

Suppose that you have a system that accurately tells you men have higher car insurance claims on average than women.

The same system might also be able to tell you people with low credit scores have higher claims on average than people with high scores.

Is it right or ok to use the first correlation and ignore the second? Or vice versa? Or maybe both of them are unacceptable? There's nothing objectively inherent in the correlations that tells you it's ok to charge men who are good credit risks the same as men who are bad credit risks. Or optimal economically! Those are independent and difficult questions.

This is tangential to the original thread; the point isn't that the world is biased. Obama's face exists in the world. The issue is that the model is biased in a way that the world isn't.

But it's completely on point, IMO: even if the model is biased against minorities in a way that the world actually is, it can be what we consider ethnically unfair.

You're worse off lending in traditionally black neighborhoods; the question is, is it ethical to make a map of all of these traditionally black neighborhoods and refuse to lend there? A model that takes this into account would be biased in the same way the actual world is, but many would consider it deeply racist and unfair.

Sure, but "there are racial disparities in this domain, and therefore in the data" and "there aren't racial disparities in this domain, but there are in the data" are very different problems. The latter is the topic of discussion and is much less ambiguous than the former.

There's also "there are no racial disparities in the training data, but the system still shows a response with racial disparity."

ML has problems with all of these at times. The particular case here with the Obama picture is a particularly vivid illustration of one that can be used to bring attention to the overall problems.

>in the same way the actual world is

I feel like you've missed at least half of what I was trying to communicate, because you're still presenting arbitrary discrimination as reality-based.

Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the true risks of individuals within the group. Because it's suboptimal, it can't be called an unique, objective reflection of the world.

All correlations are inherently imperfect and therefore unfair; if they weren't you'd have an definite causal relationship.

When you use one correlation, or set of correlations, maybe you have completely missed something that would be much better, not just from a moral perspective, but a business perspective.

If you discriminate based on gender, and haven't considered age (hypothetically it being legal), maybe the latter would be a much better proxy for the real causal factors. Your gender model can be correct in itself, and yet from a completely amoral perspective be terrible and uncompetitive. The model is not "the way the world is" just because it's technically correct in isolation.

> Discrimination as you describe is not only unfair, but it costs the lender money, because they are not identifying the people within the group who are better or worse risks.

No model is perfect. Odds are you can better predict risk by (directly or indirectly) attaching bonuses or penalties to given races, all else being equal.

e.g. Say, irrespective of all other measured quantities, people of race A are more likely to default because of other systemic racism against them. As a lender, it'd be completely rational to consider this and make "better" choices. And if you're not allowed to measure race, it'd be completely rational to find other variables that don't have an obvious causal relationship to credit risk, but predict race and thus have some information about credit risk. This is the exact kind of thing ML does.

Then, in turn, this becomes self-reinforcing. Because other institutions discriminate against race A, the risks going forward of dealing with race A increase...

>No model is perfect

But some are definitely better than others.

It sounds like you're looking at things purely from the point of view of getting the correct average for a group. But whether or not you get the correct average doesn't tell you if you're using a good enough or the best available model completely apart from fairness or justice.

If you do some type of testing and you know 5% of the tests should come back positive, is there a difference between reporting 5% at random and actually doing the tests? Of course!

But what is your definition of perfect (or better). That it perfectly aligns with societies moral view, or that it makes the company the most money.

> It sounds like you're looking at things purely from the point of view of getting the correct average for a group

No, I'm looking at things from the point of view of making a model that fits well to the original dataset and then is verified in the actual accuracy it makes over time.

If adding race-- or inferring race-- makes the model substantially better in predicting outcomes, is it right to do so? Credit default risk is correlated to race, even controlling for other variables. Hence, using race would help you make more accurate predictions.

I'd argue that curve fitting isn't modeling. And when inserting a fitted curve into a feedback system you're very likely to just perpetuate the problem you're looking to eliminate.

> I'd argue that curve fitting isn't modeling.

This entire discussion is about ML models, which can often be fairly described as very fancy curve fitting.

> And when inserting a fitted curve into a feedback system you're very likely to just perpetuate the problem you're looking to eliminate.

This is not a problem for the individual credit issuer-- they're not looking to eliminate the problem. They've avoided some credit risk by taking race into account. They've improved their expected value, even if society is stuck with the cost of the problem getting worse.

For credit reporting this sort of thing creates bad enough externalizes that it needs to be outlawed.

And I repeat curve fitting isn't modeling. Because in that case it's not a model it's a prescribed outcome.

When you say "better", better than what? How can your model tell you that some other data would not work better? The fact that you are looking at a correlation tells you that it's one of many ways to infer what you want to determine, and that it's imperfect. This is logically certain if we agree that race is not causal. So the only question is how much better is another model with different inputs.

One possible approach is to make a model that besides its intended prediction also predicts race, and penalise its ability to predict race with probability larger than random. If the model has representations that can't predict race, then it will have to use non-biased features to make its predictions. But I guess in many situations this would degrade the accuracy and it will not be welcome from a business point of view.

How would one do this? If you have a ML model with outputs classifying credit risk, and outputs classifying race, it's easy to learn coefficients that are bad at classifying race but still take race into account in classifying credit risk.

That is, what's your loss function and how does it prevent race from being considered in the credit decision?

It's a setup that is somewhat similar to GANs (and even closer to a related method called Fader Networks):

- a first network take the input data and return a representation A (like an embedding vector): let's called it the "censor network"

- a second network take this embedding A as input and is trained to predict the class that should be censored (for example the gender of a person) : the "discriminator network"

- a third network take the same embedding A as input and is trained to predict the real task of interest (for example the probability of credit default) : the "predictor network"

The idea is that, by training the censor to make the discriminator fail (predict the wrong class) while making the predictor work, it will force the censor to learn a transformation of the input data that keeps the task related information in the embedding A, but removes the information correlated to the "censored class" (and that could be used to discriminate).

Here's a reference about this kind of methods, but it's still an active domain of study in ML and there are many papers that followed this one: https://arxiv.org/pdf/1801.07593.pdf

Neat. But might it not just be easier to predict the influence of race and then use that to adjust the output/threshold?

It is not legal to make such a map. In practice, production actuarial systems for loans need to be interpretable and auditable, so this argument is moot.

In practice, multiple advertisers have been in trouble for breaking this class of rule in the past 2 years. That's not some long-forgotten problem, it's current.

If neural nets provide a way maintain plausible deniability while breaking the rules, it's going to get used.

I'm not sure neural nets provide plausible deniability. The NN can be proven to be discriminatory.

You’re missing the broader picture. This is not a problem with data, AI, algorithms or whatever. This is simply a mathematical issue that is fundamentally unsolvable.

Imagine you have two groups of people that have different means and distributions of “input” (could be intelligence, wealth, interests, ...), and this input is somehow correlated with some output (insurance rate, criminality, university acceptance rate). Both input and output are one-dimensional.

Then you only have two choices. (1) You “unify” the distribution and use it as joint input, resulting in one group (the one with the higher mean) getting “more” - many people think this is unfair. (2) Or you can subtract the means first and so make the distributions as equal as possible, which can result in a person from the “more” group having worse outcome than an equivalent person from the “less” group (if the means are close together and the distributions are wide, there will be many such people) - many people would consider this unfair.

There are plenty of examples, current and historical, of society picking one or the other, for different groups and topics: wealth/success/income (old vs young, Jews&Asians vs other Americans, men vs women, whites vs blacks), university acceptance (Jews used to be discriminated against (look up “Jewish problems”), now Asians are, men vs women [1], “positive” discrimination of blacks) - just a few. In many examples historically, we consider (1) fundamentally better than (2) - e.g. discrimination against Jews at universities - but it seems it takes the society a long time to reach this conclusion in each instance.

[1] https://en.wikipedia.org/wiki/Simpson%27s_paradox#UC_Berkele...

No, indeed, I think that's the whole point.

Approach #1 is to just go to maximize expected value, using all variables, including race.

Approach #2 is to use race to adjust the distributions as you suggest. Call this "affirmative action."

The inbetween approach #1.5, that we usually follow, is to make a value judgment about whether any individual item is OK to use.

Income? Of course it's reasonable to use income to determine whether to lend-- even if it's correlated with race. Race? No, that seems wrong. Location? Sure--- wait, you've drawn a box around all the black neighborhoods, nope! Buys certain products? Sure-- wait, you don't mean "buys products that blacks stereotypically like," do you?

#1.5 is already iffy, but becomes completely untenable once model complexity gets high and we use approaches that do not do well at explaining their decision: it basically becomes #1.

A major problem with #1 is it becomes self-perpetuating and reinforcing: if everyone is going to be biased against you, you're going to do worse, and hence the models are validated/more models are created with these assumptions. Everyone using #1 may improve their own expected value at the expense of a worse outcome for society as a whole.

> A major problem with #1 is it becomes self-perpetuating and reinforcing: if everyone is going to be biased against you, you're going to do worse, and hence the models are validated/more models are created with these assumptions. Everyone using #1 may improve their own expected value at the expense of a worse outcome for society as a whole.

Do you have any proof of that? The world is improving on all levels, even the income gap between blacks and whites has been narrowing [1].

We're not talking about 100% segregated populations like during slavery or patriarchy, the means between the groups are very close and there's a huge overlap between the distributions. It's obvious that the vast majority of outcome isn't determined by race (otherwise you couldn't have white homeless addicts and black presidents) or any such "group" characteristic but instead by "individual" characteristics (skill, intelligence, drive, effort, hereditary wealth, chance, ...), so the evidence for this "self-perpetuating" unfairness is very weak.

[1] https://www.pewsocialtrends.org/2018/07/12/income-inequality... but you'll have to manually calculate the percentages

> [1] https://www.pewsocialtrends.org/2018/07/12/income-inequality.... but you'll have to manually calculate the percentages

I'm completely overwhelmed by your evidence of median black income moving from 59% of whites in 1970 to 65% in 2016, and appearing to not change substantively in the last 15 years of the dataset. If the overall trend continues, we'll reach parity in another 269 years.

And it hardly has been happening in a world where considering race and proxies of race is considered "OK".

This is an absolutely comical piece of evidence you advance to try and justify your view.

Pricing risk accurately is the lenders' job and we certainly don't want them to stop doing it. If society as a whole wants these loans to happen, society as a whole should underwrite these loans or subsidize the interest.

I've thought about this, but it risks creating an incentive for lenders to exaggerate the risk of minority lending in order to capture larger subsidized interest amounts or externalize more underwriting risk.

I don't really understand what it means to not use race as a criteria. If we're comparing Italians and Irish, and Irish are mostly left handed and Italians mostly right, and we know that right handers are more likely to pay back their loans, then presumably my race/ethnicity blind program will favor right handers over left handers, but the consequence will be favoring Italians over Irish.

When more variables come up and get more complicated it seems like it would be impossible to say "You're based on race" versus "You're based on a multitude of factors, several of which correlate to race, and consequently you give more or better loans to race X over Y."

See https://news.ycombinator.com/item?id=23698851

>I don't really understand what it means to not use race as a criteria

You find criteria that are not inherent to race.

Reluctantly using your twisted example, imagine the left handed are more likely to default on loans because the checks they send are illegible.

Then you could use the acceptance rate of written checks as your criteria instead of handedness.

>But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets?

Yeah this is tough, answering the question of the odds a person may default on a loan will probably reflect historical bias. In fact I'd argue that it should show this if the model is accurate.

The moral issue is what do you do with that information? Do you use that knowledge to improve the outcomes of the disenfranchised or do you reinforce the bias based on the information you have?

If that is the case they should notice they are talking past each other and reset the conversation... But I guess that is very hard/impossible once it gets personal. Thus I think the fact that it became personal is the main problem...

Twitter doesn’t help finding out the misunderstanding and do a reset, you have arguments/counter-arguments spread between thousands of messages with no overview of what is being said, what are the core ideas being discussed.

A variant of this is that both ppl are answering the same question but on different time scales. So one is pro (on a generational timeline) and the other is con (on a 4-year timeline).

Both might be right, for their respective time scales, which is why it's so hard for either one to back down.

Many many ways to talk past one another. We need a small book on techniques to short circuit these situations and salvage conversations.

And that's the right way to look at it. If you can't then you shouldn't. It's not enough to build stuff and then to say 'oh, we can just fix these serious problems' afterwards if you're not able to foresee / structurally address them from the get go. Because it clearly shows a much more shallow understanding of the problem space than mere engineering can fix.

Surely this comment reaches a little too far? We have to be brave enough to try new things even if they have some risk of unknown/unintended consequences.

No decision is completely safe and no system gets everything right in the first pass.

All we can ask for is a serious effort to understand risks sufficiently well to reasonably believe that the system is safe.

You can try 'new things' in the lab, until you are sure they work and are safe for general application. To field these AI solutions in lots of places where negative side effects are a serious possibility knowing that this is the case is irresponsible. Seeing this as a simple engineering problem with a simple fix-as-you-go attitude is not the right approach.

We're talking about a university research project to rescale photos here--- not a system that is fielded.

Certainly treating PULSE as a simple engineering problem and fix-as-you-go isn't so bad, is it?

Just like 'temporary software' university research projects tend to eventually get fielded in some form or other. By not tackling this problem at the root we push the onus for correct implementation down to the people that field it who thought the academics had done a proper job in the first place with all their metrics pointing to 'success'. But success isn't a success when the dataset isn't representative and datasets used by academics as a rule are not representative or have built-in biases that are very hard to overcome by engineers aiming to field these solutions.

This has already led to numerous troubles in the applications for loans, job evaluation, criminal matters and so on.

> This has already led to numerous troubles in the applications for loans, job evaluation, criminal matters and so on.

Which I'm arguing about elsewhere... but demanding that university research about upscaling a face to be perfect to publish is a little much.

> . But success isn't a success when the dataset isn't representative and datasets used by academics as a rule are not representative

Yup, so, academics should never research this stuff or publish? That certainly is an ... interesting ... recipe for progress. And research progress is one of those things that could help us ultimately address some of these issues, because as the original argument makes clear: improving the quality of datasets is not enough.

> incommensurable

This is pretty much the issue at its core.

The real question is, if Yann is so smart, why is he arguing with people on Twitter?

He did leave Twitter though, so not a complete fail on his part.

Reply on news.ycombinator.com