Hacker News Reader

Comment by benkuhn | Hacker News Reader

I would really have liked to see them check whether the gender-blinding actually worked. Even if someone's voice is modulated, they may have gendered behavior patterns that could influence someone's performance ratings. I wouldn't be surprised if I could guess the gender of a voice-modulated person pretty accurately from other cues.

To combat this, they could have asked the interviewer during the performance assessment whether they thought the applicant was definitely male/probably male/unsure/probably female/definitely female. Then you could use voice modulation as an instrument for perceived gender and get a better estimate of the true effect of perceived gender when controlling for actual gender.

Hope they do something like that for the next round of experiments!

View on HN

Aline from interviewing.io here. Great idea!

You all should record the audio from the original modulated interviews, then play them in both male and female versions to sample sets of separate observer groups that rank the performance of the person being interviewed.

That way, you would be isolating the thing you are testing to just the voice, and you would be able to establish controls and a baseline for the exact same interview.

The way your test is described right now, there are still way too many factors that could explain the results. Your controls are far too loose.

You're absolutely right re controls. Re your suggestion, totally agree, and please stay tuned :)

I have read a lot of the comments here and your article and I find this fascinating. On controls and such: on reddit someone pointed out that more prosocial people may be more likely to sign up but I believe you controlled for this using random assignment so I think you are good there.

The biggest thing, and I believe it has already been mentioned, is trying not to bias the raters. The best way fmpov being to pre-record interviews and play them to several different people.

The above would prevent testing for attrition but I think one variable at the time is the way to go. You could test attrition later by having ratings go a certain way and then seeing how subjects respond when they know that they interviewer knows their gender vs when they dont etc.

If you have any interest in trying to get this published academically, and I want to make sure you understand I am a very lowly research assistant, I could talk to some of the professors at my university to see if there is any interest.

On a similar note, I'm wondering what will happen when somebody is perceived as gender non-conforming.

Will a woman using a modulated voice to sound male come off as a woman disguising her voice, or will she come off as an effeminate gay man? There's a lot of stigma to being perceived as the latter.

> Will a woman using a modulated voice to sound male come off as a woman disguising her voice, or will she come off as an effeminate gay man?

The sample sounded more like an Englishman to me, not effeminate per se. I've noticed a lot of Americans feel western european accents like Danish/French/English to be 'gay-ish' if they haven't heard them before. Not sure what is going on there.

Please make sure that the question is after they have already answered the other fields, and only one per interviewer. Priming is a major problem in this kind of testing, and if you are asking people to find fault in the perceived gender of a person, they then have a high chance the reverse a previous decision.

I'm not sure if voice modulation is the answer or not, but very cool to see you working on this problem and trying things out. It's a difficult space and most are springing for short-sighted bandage solutions. Best of luck, and keep sharing your findings!

This is a good point. There are often differences in how people of different genders (and nationalities, interest groups, classes, educational levels, etc) express themselves regardless of the format, and modulating their voice doesn't really hide that.

For example, I've run quite a few forums before, and I'm pretty confident I (and quite a lot of other forum admins/moderators) could reliably judge someone's gender through their manner of writing and the (non gender related) info in their profiles.

So if you wanted to really remove possible gender 'bias' from the occasion, you'd have to go much, much further. You'd have to give people a way to communicate which reduces their responses to preset answers, that's entirely conducted through text on a screen and make sure all personal information is filtered out as well.

There's always going to be a trade off here. On the one hand, I'm sure the system used in the Luigi's Mansion Dark Moon Scarescraper would completely eliminate any potential gender bias, but that's only because it limits chat to 'over here', 'good job', 'thank you very much', 'help' and 'hey'. On the other hand, an interview in person would give you a lot of information about how well they could do the job (and let you test it), but then you've got psychological biases to deal with.

It's a tough one really.

Of course completely dehumanizing the interviewer & interviewee expressly prevents any degree of culture fit selection, which could be bad for all parties.

And that's a good point too.

As much as culture fit has a really bad reputation in some places (because people seem to associate the term with 'only hires rich white kids with Harvard degrees'),you need a certain amount of it for a business to really stay cohesive and for the employees to want to work together.

I've seen companies where the hiring staff haven't bothered to think whether anyone would actually work well in that specific company and its environment, and the general result has been a heavily divided workforce with very limited communication.

Not just culture fit. I hire for a high-end (boutique) software consulting firm. We specialize in scale problems and engineering practice at a level few other firms can claim.

Communication is an absolute, 100% must for senior roles. If you can't competently and thoroughly talk about a technical issue, to non-technical people, and do it with a smile and while making eye contact, we're not going to hire you.

So in a sense, we're filtering out neuro-atypical people for senior roles. But they fit well in engineering roles.

Dehumanizing the interview process will literally make it impossible for us to gauge this skill.

Preventing culture fit is actually a feature, not a bug. Culture fit is a synonym for unconscious bias, and most people don't realize it.

This would be a fantastic additional thing to monitor for. It was my first thought after hearing the voice modulation clips-- even in a manly voice, someone saying, "Oh I totally sound like a dude" sounded quite feminine since most guys wouldn't say "totally."

Two things I noticed from the clip on the FastCompany article-- it's modulated, but I can hear A.) Vocal Fry B.) Statements that sound like they're ending with a question (the voice goes up at the end.). These are both more common speaking patterns with women. The question thing in particular is common with people who are not as confident.

What evidence do you have that some speaking patterns are more common with women? Can you link to some data?

Edit: Lots of opinions here. Still no data.

I don't know about cues in spoken English, but you can find relevant articles about written text from this page:

https://civic.mit.edu/blog/natematias/best-practices-for-eth...

Read under "Inferring Gender from Content".

I did. That section is primarily discussing analysing the pronouns used to find a gender of the target, or exploit word based genders in those languages (French in one instance) that support it.

Other examples were a selective study on Twitter, where the experimenters don't even know the true gender, or even humanity, of the user. The author of the section seems to disclaim the entire idea, somewhat (as it probably should be).

I was looking for something that could hold up to the claim that individuals of the same gender do, in fact, have common speech patterns. For example, "this pattern has only been observed with women", and, "this pattern has only been seen with men".

not quite patterns, but preferences :)

https://blog.xkcd.com/2010/05/03/color-survey-results/

Just to be clear, men do vocal fry and uptalk(ending sentences with upward inflection/making it sound like a question) too. Just less often. And not all women use uptalk and vocal fry.

Uptalk is more common in women: http://well.blogs.nytimes.com/2013/12/23/overturning-the-myt...

Vocal fry is more common in women: "An examination of creaky voice occurring in natural conversations among relatively young educated American and Japanese speakers revealed that female speakers of American English residing in California employed creaky voice much more frequently than comparable American male and Japanese female speakers." http://americanspeech.dukejournals.org/content/85/3/315.abst...

(Edit: I'd like to find better resources on this if available. These seem to be small studies.)

The NYT blog post starts out discussing regional dialects, then presents a study of 23 Southern Californians aged 18-22...

The only study the post links to is paywalled and the link to a conference paper is dead.

Your second link is also paywalled.

I've found most studies to be paywalled in general aside from the free abstract. Not really sure what to do about that.

Scihub, though I can't find a functioning mirror right now.

You could quote the relevant part of the study, or, not link to paywalls.

[It's not your fault, just the current reality.]

Unfortunately, quoting the relevant parts of a study (namely, methods and results) for the purpose of disseminating evidence is a weak fair use case at best.

That's why generally in a work you'll see "reproduced with permission" underneath figures and tables taken from other works.

The case for fair use is stronger if the work is criticism of the source material. It's a huge gray area, and some journals pursue claims relentlessly.

You're absolutely right, and I have no idea why you're being downvoted.

My best guess is that something I type triggers someone's preconceptions about a topic and then they reflexively seek solace in the down triangle.

Edit: Or, perhaps, they are rate-limited by HN and can only use the triangles?

I did not downvote your above comments, but I can see why others might. Your argument, while perhaps not intended to, parallels a standard trolling technique of asking for evidence and then dismissing any information presented. You complain about one study not being broad enough, about others for being behind a paywall, and still others for being on twitter. Ultimately, the standard you ask for ("this pattern has only been observed with women", and, "this pattern has only been seen with men") is unreasonably restrictive, as if a pattern that was only 99% reliable in separating men from women would not be sufficient. It makes it appear (again, this may not be true, it's just the appearance) that you're not making a good-faith effort to evaluate information fairly, but are instead holding firm to a preconception and requiring extraordinary proof before being willing to reconsider. Your snarky "triggers ... preconceptions" response does not help this perception (and, FWIW, I did downvote this comment, because your "guess" is of negative value -- it only serves to upset your supporters and anger your detractors.)

EDIT: also FWIW, I would love to see good data about potential differences in vocal patterns in males and females, if for no other reason than that I could use it in my D&D campaign.

You seem to be confirming my guess (perceiving a "standard trolling technique" === "preconception"), but I am not looking for supporters or detractors, only justification for the original claim:

>These are both more common speaking patterns with women.

It is incorrect to ask for "only observed" data in this instance, but I also was clear to state that it was only an example of data that could be presented.

>parallels a standard trolling technique of asking for evidence and then dismissing any information presented.

If the response is not actual evidence, it should be dismissed. There is no guarantee that a response to a request for evidence is actually valid. That requires investigating the claimed evidence. In this case I investigated and found it severely lacking, as I noted above. Another user has linked to some further analysis of one of the claimed pieces of evidence (the 23-person Southern California study) that shows an apparent basic arithmetic error in the presentation of results. The NTY blog dead links that study and other places that have it require one to pay to see it.

>about one study not being broad enough

Yes, the study was 12 women and 11 men, aged 18-22, living in the same geographic location. There were severe differences in the experimental conditions between the groups being studied (women all interviewed by women; most, but not all, men interviewed by men). Everything I have read also indicates that this study found that both the men and women in the study often exhibited the pattern that was being watched for.

>and still others for being on twitter.

The problem is not about being on Twitter, but that the accounts being studied were neither 1) confirmed to be human, nor 2) confirmed to be the gender the experimenters assumed (and to which they compared their predictions/guesses! [in-built confirmation bias?]).

>that you're not making a good-faith effort to evaluate information fairly, but are instead holding firm to a preconception and requiring extraordinary proof before being willing to reconsider.

I can't evaluate what I can't see and I don't see how I have been unfair to anything that has been presented here. I don't think I am asking for extraordinary proof. I'm only asking for evidence of the particular claim:

>These are both more common speaking patterns with women.

So, something like: randomly sampled recordings from public places with counts of the occurrence of the pattern and the gender of the person who spoke it.

>> "(perceiving a "standard trolling technique" === "preconception")"

You claimed preconception about the topic. Perceiving trolling is a preconception about your rhetorical technique. Don't overlook that important nuance.

>> [SNIP -- attempts to justify]

I don't care whether you are or aren't justified in rejecting the data (I have no dog in this fight.) I'm simply noting that you seem to give off the impression that you're not genuinely interested, or open to being persuaded. This is more than just whether or not you accept the studies; it's in the phraseology you choose. Just as an example, you found the time to be snarky about downvotes, but not to express gratitude toward any of the people who have tried to provide you with data (even if the data is insufficient.)

> "I don't think I am asking for extraordinary proof."

The line I quoted ("this pattern has only been observed with women") was an unreasonable, extraordinary standard of proof. In a vacuum I would have ignored it. But in combination with the snark and dismissiveness, the sum total does fit a pattern that is common to people trolling on all sorts of topics.

Again: I don't have a dog in this fight. I'm just trying to help you understand what about your behavior might be causing people to perceive you as worthy of downvotes. I'm trying to give you insight as to how, if you are interested in genuine dialog, you can make that clear to others. Because your posts thus far create a very different perception than that.

I claimed preconception about a topic. I do not know which; I only assume that a downmodder had something in mind. I suppose it's possible they are completely random downmods from wandering bots.

> I'm simply noting that you seem to give off the impression that you're not genuinely interested, or open to being persuaded.

You're not the first person to conflate me writing and giving attention to something as being "not genuinely interested". I honestly do not understand this perception, particularly when I am discussing details of things brought up deeper into a conversation. Why would I waste my time on such trivialities?

>The line I quoted ("this pattern has only been observed with women") was an unreasonable, extraordinary standard of proof.

You'll note that line is not in this particular thread. It is however related and I agree that it was poor.

>I'm just trying to help you understand what about your behavior might be causing people to perceive you as worthy of downvotes. I'm trying to give you insight as to how, if you are interested in genuine dialog, you can make that clear to others. Because your posts thus far create a very different perception than that.

Thank you for writing out your perceptions, even though you only downvoted one of my posts. I don't think the perceptions you describe as others perhaps having accurately characterize me, though. I am not attempting to create a particular perception; I'm only trying to have an on-topic discussion: an absolute claim was made that seemed to lack observed evidence -- and despite many posts discussing it now, no real evidence has been shown.

> "... conflate me writing and giving attention to something as being "not genuinely interested". I honestly do not understand"

There is a classic trolling technique, which I have heard called a "snow job" but probably has other names, wherein someone tries to basically overwhelm a topic of discussion with minutia in order to get people to waste their time while not actually making any progress in the discussion.

Relatedly, a common technique among people who are dogmatic about a position is to build an impenetrable wall (built out of unreasonable standards of proof, and nitpicking criticism) and then spend a lot of time arguing about how their wall hasn't been penetrated. In essence it becomes a way of inflating the ego -- "we argued for hours and the other guy never even put a dent in my position! That's because I'm so smart!"

Again, not saying this is what you're actually doing, just noting that it's easy to get that perception from reading along.

> "I don't think the perceptions you describe as others perhaps having accurately characterize me, though."

Perhaps they don't. But the first comment I made in this subthread has a score of +10 right now (an unusually high score for a meta-comment of this sort, particularly this deeply nested), which indicates that quite a few people share the perception.

> "I am not attempting to create a particular perception; I'm only trying to have an on-topic discussion"

Whenever you speak or write, you create a perception. Whether intentional or unintentional, it affects your ability to have an on-topic discussion. My hope is that you'll be able to take what I've written here and learn how to reshape that perception, in order to more effectively communicate your interest in an on-topic discussion.

One final thought: the comment "no real evidence", likewise, contributes to the perception. What has been shown might be weak evidence, but it's not nothing. In a Bayesian sense, if you started with a neutral perception ("I do not know if there is a significant difference") what has been presented would be suggestive that a difference might exist, but not strong enough to show that a difference definitely exists. The proper response would be "this seems like an area where better data could be gathered" rather than "no real evidence has been shown".

That second study aside, which I cannot access the data to see how well the conclusion is supported, no other study that I have seen says that vocal fry is more common in women.

In men we generally don't even notice it. It's just the way men talk. Here are some videos of actors all with loads of vocal fry, but, except when presented as a list of "men with vocal fry," no one would ever think to pick up on it and mention it: http://the-toast.net/2015/07/22/examples-of-male-vocal-fry/

Vocal fry is more commonly noticed and criticized in women but I haven't gotten the impression that there's definitive evidence that women do it more than men since most of the studies have been pretty small or localized.

Mark Liberman has been covering this for years over at Language Log and there appears to be a great deal of ongoing debate within the academic community:

http://languagelog.ldc.upenn.edu/nll/index.php?s=%22vocal+fr...

http://languagelog.ldc.upenn.edu/nll/?p=20155 discusses male usage and briefly mentions the study from your second link.

http://languagelog.ldc.upenn.edu/nll/?p=13020 has some graphs of what exactly this means sound-wise and http://languagelog.ldc.upenn.edu/nll/?p=3626 discusses the history.

Thanks for this. I was having trouble finding anything discussing vocal fry in depth without having to pay. I see numerous articles on women doing it more, but so far they all seem to point to the same small studies.

How can a gendered speech "fashion" (which is what it seems like to me) be criticized? I think it's hot (as a guy)... probably because it's so sex-typed

It's usually mentioned in a negative context as something young women recently started doing and should stop.

http://languagelog.ldc.upenn.edu/nll/?p=3626 covers the claims that this is some new trend, many of which are couched in language which implies that kids these days are doing something wrong or degenerate.

http://languagelog.ldc.upenn.edu/nll/?p=12774 discusses media coverage a couple of years ago claiming that women's careers (or vocal cords) would be damaged by using vocal fry.

This American Life did an entire episode (http://www.thisamericanlife.org/radio-archives/episode/545/i... – http://languagelog.ldc.upenn.edu/nll/?p=17489 has discussion) about the volume and vehemence of the complaints they receive about female cast members' use of vocal fry:

> Listeners have always complained about young women reporting on our show. They used to complain about reporters using the word like and about upspeak, which is when you put a question mark at the end of a sentence and talk like this. But we don't get many emails like that anymore. People who don't like listening to young women on the radio have moved on to vocal fry.

> What's striking in the dozens of emails about vocal fry that we've gotten here at our radio show is how vehement people are. These are some of the angriest emails we ever get. They call these women's voices unbearable, excruciating, annoyingly adolescent, beyond annoying, difficult to pay attention, so severe as to cause discomfort, can't stand the pain, distractingly disgusting, could not get over how annoyed I was, I am so appalled, detracts from the credibility of the journalist, degrades the value of the reportage, it's a choice, very unprofessional.

The part which underscores that there's a strong gender-based component to criticism is that Ira Glass, the male host of This American Life, is basically the king of vocal fry (along with other long-time radio personalities like Carl Kassel or Walter Cronkite) but nobody complains about his usage and on the rare occasions when any mention is made, it's not expressed in the same kind of negative language:

http://languagelog.ldc.upenn.edu/nll/?p=17496

Well, now that I know about it and the name of it, I won't be able to un-hear it... which will probably make it "annoying," when previously it was just "young-womanish", sigh. ;)

I'm assuming that all of the complainers knew the name of it and of its existence to begin with. I wonder if there's a negative preferential reaction there.

Like many things, this is a taste, and the same taste can seem delicious to one person and disgusting to another. I know someone who can't stand chocolate...

dude, what decade are you from? I'm totally stoked about this totally tubular voice modulation. It's like totally wicked dude! Totally!

What if you just used a third party, like a translator? Someone listens to the voice and recaps it in his own words.

In the clips the modulated female ended up sounding like an effeminate man, which raises a whole other set of questions about bias.

There should have been a control experiment where a person's likeability, intelligence and "hirability" are assessed when both women and men say the same things with and without modulation to make sure it is really neutral.

I think it would be amazing if they could combine this voice-modulation technology with a facial motion capture technology (like in Avatar) to display artificial/shifting faces as well as voices. You could effectively create a computer screen-based version of the "scramble suit" used in A Scanner Darkly! http://www.imdb.com/title/tt0405296/mediaviewer/rm1872861440

I think a combined face+voice system would be even more useful in interviews (since a significant amount of human communication is expressed through facial cues). It would also add another dimension to your proposed voice-modulation gender-blinding experiment.

Wouldn't that just increase the problem that the person you are responding to brought up? If you capture their facial movements, it might be even easier to tell if they are a man or a woman, despite any scrambling you do.

Quite possibly! That's the extra dimension added to the experiment. :)

It would be even more fascinating if you could cluster facial expressions or motions as "more female" or "more male". And for bonus points, work out a way to "filter" or "smooth" the facial signal to reduce/remove these giveaway hints (or insert false hints).

Yes. If the goal is gender blindness, a more effective strategy is to block all video and voice communication. Even chat could be a giveaway. Go off of nothing but code against a test suite.

That sounds like a recipe for disaster. Coding skills that can be measured in this way are a small part of what it means to be a good employee.

I feel like this is kinda like choosing who to draft in the NBA solely on free throw percentage.

IIRC studies have indicated that (be it for genetic or cultural reasons), women tend to be more facially expressive than men.

In the example clips, to me, her vocal fry was a gender tell, even after modulation.

Vocal fry isn't gender specific.

Ira Glass (This American Life) has vocal fry and they discuss the gender bias here: http://www.thisamericanlife.org/radio-archives/episode/545/i...

What if Ira Glass isn't a Kinsey Zero? https://en.wikipedia.org/wiki/Kinsey_scale

I actually don't think you should put the question on the interview feedback form. That invites bias into the question. "Oh, this candidate did well, must be a man." In fact, having interviewers consciously guess at the gender may even skew their perception of the interview performance post-hoc.

I would, however, be interested in a separate experiment to gauge the effectiveness of the voice modulation.

Ask after the normal survey is complete and submitted.

Only works once per interviewer.

You can't prevent people from asking themselves if a modulated voice is a man or woman. If they are inclined to ask that question, then they are going to ask it no matter what you do.

Correct. What you can do, however, is prevent their answer to that question from being better than a coin flip. This is why we should run a separate experiment to gauge the effectiveness of the voice modulation.

To provide a specific example, some women have severe vocal fry / creaky voice. Its a socially transmitted thing like valley girl speak from decades ago.

I can only imagine how weird that sounds autotuned in a deep bass artificial voice. Imagine the movie line "I'm Batman" with vocal fry.

(edited to note, I see three people simultaneously cited vocal fry as an obvious experimental error and one tried to derail with the standard citation needed technique, so its probably a pretty serious issue)

Men speak with vocal fry all the time.

Listen to these interviews. All the well-known actors in them are speaking with tons of vocal fry -- much more than "Valley Girl" women ever would.

Except we never even notice it in men, except when it's presented in a "list of examples of male vocal fry."

http://the-toast.net/2015/07/22/examples-of-male-vocal-fry/

While not scientific, I heard a pre-recorded interview done with interview.io's gender masking where both sounded like men (in reality one was a man and one was a woman). I got the genders wrong when prompted to guess, so it was pretty damn convincing from my POV.

"Like, Totally!"

If you're tracking the interviewer gender guess, that's a legal liability to the interviewer. I'd refuse to answer, or always put "unsure" even if I was 100% sure of the gender.

Translation: "Hope they keep doing this experiment until it matches my expectations!"

Reply on news.ycombinator.com