The Impact of “Blind” Auditions on Female Musicians (1997) | Hacker News Reader

The Impact of “Blind” Auditions on Female Musicians (1997)

nber.org

52 pointsqiqing52 comments

Someone should build an job search/interview site that gives "blind" interviews: scrubs candidates (application materials, resumes, etc.) of gender, race, age, etc. You might not be able to blind the whole process, but you could blind the preliminary steps. Even that would probably effect the composition of the final selections, as it did here.

I had a similar idea: "Meritaco- a double-blind git middleman that hides identifying information from pull requests."

A similar thing has been done many times by submitting the same resumé with male and females names, or with names statistically more likely to be white and names statistically more likely to be a person of colour, with different ages, and so forth.

Every time they run a test like that, they discover that who you appear to be matters more than your objective experience.

I know these studies are common (probably because they are easy to conduct) but they don't measure the intrinsic bias of the interviewer.

What they measure is a combination of the intrinsic bias of the interviewer, and the statistical inference that the interviewer does based on race/gender (statistical discrimination).

Because a resume is a noisy signal, it is still possible that race/gender contains extra information even after you have seen the resume. For example, suppose people tend to exaggerate, and this exaggeration introduces some randomness. Given the signal "lead some impressive project", there is some probability that the person didn't really lead the project. Now if the probability of exaggerating is the same, but a Black person or woman was less likely to have lead the project a priori, then even after observing the resume that claims to have lead the project, the a posteriori probability of having lead the project is lower for the Black person or woman.

how do you not understand that "statistical discrimination" is a subtype of "intrinsic bias"?

Many people, perhaps not you, are interested in distinguishing between discrimination based on Bayesian inference, and discrimination that isn't, and is instead, for example, based on incorrect views or inherent dislike of certain groups. You might say that both are equally bad. I suspect that if you were given the opportunity to argue that a particular form of discrimination is not justified by Bayesian inference, you would do so. It's only because I'm suggesting that it's possible discrimination is justified by Bayesian inference, that you are (rudely) objecting to the distinction.

You say Bayesian inference, I say bigotry.

I think unfortunately Bayesian inference often does mean bigotry. And bigots aren't necessarily always wrong.

I think what would really help these kinds of discussions is to first agree on whether we're discussion a 'should' or an 'is'.

In the latter case, there are many conclusions one can draw that are considered racist, misogynist, or discriminatory in any other way. And this knowledge can be valuable for study, or other ways.

In the former case, however, all of that doesn't necessarily matter, partly because we assume the reasons for things being as they are, are a result of unfair processes, and partly because we prefer to give the individual the benefit of the doubt over trusting on the statistics that would put this individual in a group that he might not really be part of.

I often feel that both types of discussions are significantly harmed by conflating the two.

I thought (and it occurs to me I have no evidence for this and don't know where the belief came from) that probability of exaggerating was lower for women and black folks.

Responding to your highly inflammatory hypothetical:

You have gone from a nostrum about an entire population ("...a Black person or woman was less likely to have lead the project a priori..."), which could have included thousands-to-millions of people, to a statement about one particular individual.

The grossest error of this way of thinking is that it is mixing a vague, dubious, and unquantified signal (your a priori "knowledge") with a very high-quality signal (a specific and verifiable statement made by a single person about a single project).

If you're really proposing to do some kind of "Bayesian" weighting of these two pieces of knowledge, you're trusting your machinery for assessment of probabilities way too much. That a priori knowledge is junk compared to the statement on the resume.

Or, to look at it the other way round: If you're so well-calibrated that you're taking population-wide information into account, I shudder to think what you must be doing with other side information like the font, page layout, semicolon count, or paper composition. Lump it into the prior! What could possibly go wrong?!

I must add that you're deploying a hyper-logical argument in a real-world situation in what is honestly a stupid fashion. Nobody who does real-world inference should operate this way.

We are talking about academic studies, not how I would personally act.

You are asserting that the signal from race/gender is very noisy and the signal from the resume is very precise.

We can debate the precision of the signal from the resume, but race at least is highly predictive of many objective qualities, e.g. it is highly correlated with IQ. So what you call a vague, dubious, and unquantified signal is actually a highly informative signal.

I notice you went right from race being correlated with IQ to race being predictive of IQ. Unsurprising.

This is a discussion on probability and statistics, so I was using the technical terms. In statistics if A and B are correlated, then A predicts B. But you're just a liberal who assumes everyone who isn't is dumb. Fuck you.

Hey, there's another thing that's very, very predictable: That a person whose argument is making bigoted remarks and claiming they're neutral statistical results very quickly devolved into saying 'liberals are stupid!'

> You are asserting that the signal from race/gender is very noisy and the signal from the resume is very precise.

> We can debate the precision of the signal from the resume, but race at least is highly predictive of many objective qualities, e.g. it is highly correlated with IQ. So what you call a vague, dubious, and unquantified signal is actually a highly informative signal.

You’re only making a very short statement, so I don’t know what you personally think. However, the statement is imprecise enough that others may mistake what you mean for the following fallacy.

Let’s say we have two kinds of Sneetches: Those with stars and those without stars. A star is highly correlated with success taking a certain type of test that we’ll say measures “Scintillence.” I am interviewing Sneetches for a job where scintellence is also highly correlated with competence. I ask for Sneetches with five years of experience doing this job.

Now: Should I refuse to interview Sneetches without stars, because not having a star correlates with less success in the scintellence test, which then correlates with less competence in the job?

The trap that many fall into is saying that since there is a correlation in the general population of Sneetches, we can draw inferences about the Sneetches applying for this particular job. However, we are dealing with the subset of Sneetches who have already demonstrated their aptitude for the job by having five years of actual experience competently performing a job that correlates with scintellence. We are not selecting Sneetches at random from the general population, we are using a combination of self-selection (“apply for this job if you have a desire to do this job”) and external filtering (“apply for this job if you have five years of experience doing this job.”)

The presence or lack of a star on a Sneetch may be highly informative about their ability to do this job if we pick Sneeteches at random, but that’s not what we’re doing here, so no, it isn’t highly informative for the purpose of choosing whom to interview.

Summary:

The presence or lack of a star may be highly informative if we have no selection pressure on the sample, but when we apply other filters that are themselves correlated with the attribute that interests us, it loses its ability to inform us.

> I shudder to think what you must be doing with other side information like the font, page layout, semicolon count, or paper composition.

You joke, but one of the best predictors of being accepted to (a particular) graduate business school (while I was still working in admissions) was to simply look at the style, formatting, grammar, etc of their resume.

It's likely that with a large enough corpus, you probably could extract some meaningful signal out of just that information.

I'm aware of those experiments. I'm not talking about an experiment. I'm talking about an actual online job interview service ... I'm not sure if there is anything analogous now, basically the next step after a LinkedIn or monster.com, that blinds as much of the interview process as possible to minimize these biases.

One problem I have with this is that technical interviews are not as straightforward as orchestra auditions. Technical interviews are already argued about frequently and some important qualities in a candidate (culture fit, how easy they are to work with, how they handle pressure, etc) are hard to determine through a blind interview. As you said, there would have to be non-blind interviews anyway. I know many companies already have preliminary online coding problems, but once the candidate gets to the meaty in-person or phone interviews, we are right back where we started.

Additionally, many of the standard algorithmic questions are similar to UIL and ICPC questions which are dominated by males for various reasons. Even if we could have completely blind interviews, I don't think that a significantly higher percentage of women would get the job because of larger problems in the tech industry like convincing minorities to join clubs like UIL and ICPC or enter the tech industry in the first place. That being said, I still think this is an interesting idea. I would definitely like to go through a blind phone interview to see if the interviewer treats me differently. Bring on the voice changer.

"Culture fit" is a bafflingly backward bit of bigotry hiding in plain site. "Culture fit" allows companies to discriminate against a wide range of characteristics. I'm amazed any company thinks it's acceptable to include culture fit as part of the recruitment process. (Although I'll grudgingly accept it might be part of non-retention).

It seems "Culture fit" often just means young and willing to put in extra hours in exchange for foosball, snacks, and team building at the local bar.

Yes, and those exclude many people some of who have protected characteristics.

Thomas, Erin, and I are working on it. (It's a collateral goal of Starfighter.)

I wonder if Starfighter can do something like that. At some point they're going to have to give real info to their clients but they could mask these details as long as possible.

If you're responsible for choosing applicants to interview from a stack of resumes, you can do something similar by simply having someone else strip the names out before you see them.

While you probably are not consciously racist or sexist, simply consuming mass media and living in the society we live in gives us aliefs (subconsciously held beliefs that we consciously know are wrong) that can affect our choices without our knowledge. It's substantially harder to remove your subconscious from the equation when conducting interviews, but at least choosing the best candidates to interview is a step in the right direction.

Which is why I'm always puzzled as to why job applications ask for information that has no bearing on the process of qualifying an applicant, such as age, gender, nationality, race and so on. Unless you are specifically looking for someone of particular age, gender, nationality and race why ask for this information?

In theory that information is used by the company to male sure they are not discriminating against any groups. Ideally that information is on a seperate sheet which is never seen by people in the recruitment process; it goes directly to HR to use for their stats.

OT, but that's a funny slip:

"... by the company to male sure ..."

Job applications do not ask for that information. It's illegal.

This depends greatly on local law.

I'm always amazed that places like Germany generally have people including photos with their resumes. Blew my mind when I moved there from Canada.

> Female musicians in the top five symphony orchestras in the United States were less than 5% of all players in 1970 but are 25% today.

The article was published in 1997; it would be interesting to see some more recent numbers. I searched briefly and couldn't find anything definitive.

This seems to suggest that when we know someone is a woman, we judge them more harshly.

To me this kind of thing begs questions around affirmative action and other methods of non-'absolute-performance' admittance to jobs/schools/etc. I totally agree that for one person to get to the same 'absolute-performance' point can mean that different people had to come a lot farther than others, but this article implies that it doesn't(shouldn't?) matter. In college admissions though, its never colorblind, socioeconimically bling etc. I'm never really sure I know where I stand as on some level the absolute performance for a job is what matters to share holders, other employees and even end users - but as a society, the idea of whole groups of people being left behind seems shitty.

Affirmative action in college admissions serves a different purpose; it's explicitly not a pure merit-based evaluation, and really is poorly understood.

A sort of basic understanding of how to implement it would be to assume you have a points-based system for admission, which takes into account things like previous grades, standardized test scores, etc., and normally has a minimum threshold of n points for admission.

One way to think of affirmative action is that it says "people from this group, on average, will score k points lower not because of a lack of ability or intelligence, but because of a lack of opportunities earlier in life due to social inequalities", and then asks questions about whether a score that is within k points of n should be admitted.

The real implementations are more complicated, of course, but this is the basic idea. The main problem with it is that starting to apply remedies for inequality after someone has already lived through 18 years of it -- and 18 important, formative years at that -- is nowhere near enough.

The best way I've heard it put is that the admission process seeks to measure the "velocity" of a potential student and not just position. If someone has come from much further behind than another, they may be a better candidate for the school despite being at a slightly lesser relative position at the time of application.

And has it been evaluated in this regard?

Yeah. Almost certainly not. But the woman who said this to me was obviously a true believer and at the time it touched my geek sensibilities far more than the cynical vision of a pantone swatch of skin colors with hash marks behind them required to maintain state funding.

Some things you just really want to believe.

Your explanation doesn't account for Asian Americans.

There's been plenty of affirmative action related to Asian Americans in university admissions. What often happens is that strict reliance on grades and test scores gives an oversupply of people who are of Asian descent or Jewish and when the university's management and alumni see that, they push for affirmative action to get the white Christian men percentages up to a high enough number that the largely white Christian men of the university's management and alumni feel comfortable.

In what way is affirmative action poorly understood? What you have described is simply the bar being lower for certain groups, which is exactly how affirmative action is understood by most people.

>One way to think of affirmative action is that it says "people from this group, on average, will score k points lower not because of a lack of ability or intelligence, but because of a lack of opportunities earlier in life due to social inequalities"...

Which is why the people actually helped by affirmative action tend to be middle class people from favored ethnic groups who had opportunities.

"Affirmative action" is a terribly divisive, blunt tool which creates more injustice than it remedies. The idea a poor white guy from a broken home in Appalachia is somehow starting out behind a middle class black girl with an intact family in Southern California is laughable.

I've always thought if you were going to have any kind of affirmative action, it should be class/income based rather than race and gender based.

Sure the most powerful people in the world are usually white men, but as you said that doesn't really mean much to the average poor rural white kid.

Also income is also highly correlated with race, so disadvantaged minorities would benefit disproportionately anyway.

"socioeconomically bling"

I'm not sure what this typo should mean, but I do know it needs to become a central phrase in American culture.

Also interestingly, this study perhaps shows that there isn't a bias towards hiring women in this case for "political correctness". The screen prevents this.

This study was published in 1997 though. I'm not suggesting there is a bias problem for "political correctness" sake nowadays, but I think the people who would argue that there is wouldn't accept a study from 18 years ago as a counterargument.

It'd be interesting to see a similar study done in 2015, for this reason, and also just general comparison over time.

Sure! However, I feel like the grand irony of these kind of studies is that the same kind of unconscious selection against orchestra members is probably also occurring in evidence acceptance. It feels like people are overwhelmingly critical towards evidence that women are treated unfairly in a way they aren't to other evidence. But hey, that could just be a feeling.

Isn't the exact opposite also possible? That there are all kinds of biases in the world, but studies are being selected for (a) showing a bias against women and (b) showing that women are objectively as good as or better than men?

Sure! But I'd think that based on history there's a fair chance that studies that show (a) are being selected for because they are factual. For example, zero women have been american presidents (although maybe that'll change soon) :). Also there is a lot of evidence that the historically assumed inverse of (b), that man are as good as or better than women, in almost every domain, has made the selection of (b) something that people immediately perceive as an anomaly, a bias. Really, it makes sense that women are better than men at some things on average, and that men are better than women at some things on average. The number of women in tech certainly shows that there's some things at least that women aren't being hired as much for. I'll have to see the data suggesting the selection of "facts" is now leaning entirely towards women, but I haven't.

To put it another way, the following article was posted, upvoted, and generated almost no discussion regarding its "methods" and the "evidence" it presented:

https://news.ycombinator.com/item?id=8909954

Is it possible that this is a result of there being a bias against the evidence presented in pro-woman/anti-bias articles, while pro-man articles receive little attention? Have we sufficiently offset that bias?

It did generate some discussion before.

https://hn.algolia.com/?query=Is%20There%20Anything%20Good%2...

The original study does not appear to have had any control groups, unfortunately. There are too many possible confounding factors present to be able to attribute any of the difference definitively to bias.

Reply on news.ycombinator.com