Hacker News Reader

Comment by throw10920 | Hacker News Reader

> diversity & inclusion review

Is this tongue-in-cheek, or are you serious? Poe's law and all that.

If you're publishing a dataset in the terabytes it does actually make sense to at least do a pass over it and make sure the data you're using isn't skewed in any undesirable way that would cause problems down the road. For example, if you're releasing 5tb of face photos for training facial recognition nets, it would certainly be a problem if all the faces are white women or asian men - the result would probably be over-fit and not perform as well for people in other categories. It would be correct to call that a diversity/inclusion issue.

Privacy and accessibility reviews serve similar purposes there, you're reducing risk by checking for these various problems and ideally they also spot ways to improve the quality of your outcomes.

the 5tb was performance data collected from servers

Sounds like the reviewer would glance at it for 5 seconds and say 'ok'

What if some servers were excluded?

It's common in fintech for data/ML models to go through similar overview. If you happen to disenfranchise a set of people because your model said not to lend to them, you risk legal jeopardy.

To clarify, I think it's good that this is a practice.

A review doesn't necessarily mean you need to resolve all diversity/inclusion issues. It can merely require that you identify the issues and understand the risks of not resolving them.

The whole point of the model is to find who not to lend to. You are always going to exclude people by definition.

Tell that to the legislators and prosecutors who create laws and enforce laws against you.

There are so many ways you can accidentally systematize racism in software like automated lending.

In the past there were explicitly racist policies like redlining. This leads to a historical data set of loan denials to people in specific racial group. If that group has other traits that correlate to their race, e.g. the neighborhood they live in then you could presumably have a model that doesn't explicitly have race as a feature but uses that historical data and some subset of racially correlated features and as a result disproportionately excludes people of that race.

I am not sure how one would remove all ageism, sexism, racism, classism, title-ism, and so on from lending. The whole concept is about making a prediction about the future with sub optimal information, guessing who will default on a loan and who won't. Same goes with insurance.

I have been pretty tempted to lie about where I live in order to reduce my insurance costs. It would reduce the insurance cost by half. It seems pretty disproportionately harsh that I should get lumped together with the people who simply happen to live around me.

Is it possible to make predictions illegal if they are based on historical data from other than the individual customer?

Yes, but we should exclude people for valid reasons, not for their race.

I should clarify, the point is to not discriminate against a protected class.

Partly tongue-in-cheek. These review processes exist, but whether they're required or not depends on the product area and type of project.

Perhaps Google don't want to be in the news for identifying dark-skinned people as monkeys again?

I can't remember which company it was that launched a camera with face identification features, but that didn't recognize any face that wasn't lilly white like every single engineer that worked at that company. They could have probably benefited from a diversity and inclusion review. Heck, employing a single brown engineer or even QA engineer probably would have been enough to notice that before launch.

> I can't remember which company it was that launched a camera with face identification features, but that didn't recognize any face that wasn't lilly white like every single engineer that worked at that company

It was HP https://www.youtube.com/watch?v=t4DT3tQqgRM

People jump to the racism conclusion here, but it's really just a contrast issue.

Detecting eyes, for example is simply easier with lighter skin.

Light skinned black people read just fine, and super tanned white people are harder to read. It's literally contrast (light) detection, not racism.

But because the media keeps everybody primed for racism to stave off the necessary class power rebalancing, everyone jumps to racism.

It may be unintentional, but it shows that they didn't test with anyone with a darker skin tone, which shows the biases at work.

It doesn't show that. It's literally numerical in that dark skin reflects less light than light skin , so the sensors report lower values for the entire face, reducing contrast for the entire face, which is what the recognition systems count on.

Brown eyebrows on brown skin = low contrast.

Brown eyebrows on pale skin = high contrast.

If our races were dark purple hair on bright green skin and bright green hair on dark purple skin, facial recognition systems would have no trouble with either. But that's not how humans render, so our contrast based systems struggle with low contrast.

It's like you're confusing a software/data problem with a photon/physics problem because you're thinking in your box.

It's a design problem. If they tested it with POC they would have noted down "well, our primative algorithm works well for light skinned individuals but not others"

And hopefully someone wouldn't have said "hmm good enough for me, let's ship it!"

>algorithm

This just proves that you're assuming it's a software problem when it isn't.

This is basic GIGO. The light sensors feed poor quality data in for people with low contrast faces, so there's nothing the software can do about it.

Couldn't it brighten the image in some way?

My webcam has an advanced option panel that lets me edit both the brightness and the exposure time. I can turn it up so bright that you can't even make out any of my facial features, and I'm in a somewhat dark room lit by a single floor lamp.

Here’s a link to an article from the time with an expert on the subject explaining that it’s not a contrast problem:

https://www.baltimoresun.com/bs-mtblog-2009-12-hp_racist_web...

Here is what I said:

> it shows that they didn't test with anyone with a darker skin tone

Are you disagreeing and saying they did test on people with darker skin tone, found the issue, and decided to ship anyway? You realize that either way, it doesn't make them look good?

Anyway, leaving all that aside, the article interviewing an actual face recognition software expert, shows that your guesses here are incorrect.

They aren't guesses. It's physics.

It’s only a “contrast issue” if the people building the system failed to have roughly half of humanity represented in any meaningful way on their dev team.

They need to test on a realistic sample of users. Testing on the dev team is just lazy; they probably have unusually new and expensive hardware in well-lit offices.

There is really no reason why the dev team should include any particular demographic: how are you supposed to have 90 years old people in the team to make sure they are recognized correctly? This is a requirements issue which directly impacts validation/test data collection. If their user base has 50% black people any reasonable protocol will include enough black faces int he test data to detect the problem early on. Ml based systems will always make errors, which errors matter will be defined by market/legal/mission requirements. It may very well be that faces of black people are harder to detect (especially in backlit situations). Should you hold the product because it may not work for everybody? It’s a complex decision. Maybe you can just have a good “face detection failed” flow to handle all the errors (think not only black people but also, tattooed people, etc.).

Arguing that having quotas of that or the other in the dev team will make them more sensitive to diversity issues in general is also unnecessary because everybody is part of some minority in some situation, hence a minimum of education will make anybody understand first hand the value of inclusiveness and diversity.

Btw, the team is using only their faces to test the system they won’t go far.. (think about lighting condition / different environments).

> Ml based systems will always make errors

Sure, but error should be randomly distributed. This is stats 101. Any decent ML practitioner will check for this before releasing a model.

In theory I agree with you, we want unbiased models but here we have an input distribution that is not well understood so things get much more complex. We don’t even have a clear definition of what’s a face or not.

The model doesn’t work for people with masks: near 100% failure rate on this category of inputs. Should we release it or not?

In general some inputs are harder than others so it is expected to have more errors on those.

That being said in practice, in normal conditions, it is not hard to detect people with dark skin if the proper training data and training is used (btw, if you don’t pay attention how you do things even a low light image of a Caucasian will not be recognized) so there is little excuse to exclude a large part of the population just because of sloppiness. Moreover for this specific category (and of course others), there are consideration ethical and legal to make sure the system works for them.

Apart from that in general I do really think that ML systems with no “operator override” in many contexts are an hazard. We cannot expect the model creators to have predicted and tested for every possible input and we cannot have ways to manually correct the error (for instance in lending or border controls). Incidentally it is interesting to note this will be skilled work that will not be take over by “AI”.

I believe we're mostly in agreement. What's not acceptable to me is using "All models are wrong" to imply that it's ok to not understand ways in which they wrong, to be willfully ignorant of their failures, or to devalue transparency.

As a professional and practitioner, I have to a responsibility to engage in transparency and honesty when I deliver a model. Part of that is understanding and designing failure modes. That's simply good engineering.

Indeed I agree, it seems even that for some use cases training data is not anymore the bottleneck but robust test suites are. Interesting times, let’s hope we will find a responsible way to use these powerful technologies.

Contrast may be one root of the technical problem, but claiming a product "ready to launch" while it fails to work for people based on their race (especially when the company clearly didn't put effort into preventing the issue ahead of time) is problematic.

By having a diverse team (or making some effort to include diverse opinions) you'd have a chance to discover new ways to detect faces, or new mitigations to the contrast problem.

But claiming a product is ready for release when it excludes people based on race (no matter the technical reason) is a problem.

And yet a short while later, they released a patch that fixed the bug. So your physics claim is irrelevant.

The fact remains that they would not have released that software knowing it wouldn't work for Black people. And yet, they didn't notice the bug because they were making no effort to be inclusive.

Microsoft Kinect also had this issue. I know the one black engineer (that had little to do with the project) that repeatedly got pulled in to see if the test system worked with non-white people.

He left soon after.

I wonder if the company failed to give him credit, responsibility, or compensation commensurate with his value to the project?

Reply on news.ycombinator.com