The list of disabilities that the instruction to not be ableist affected and didn't affect is interesting. "Disability", "deaf" and "blind" (in that order) become the first choice >=80% of the time. The instruction moves the needle on "autism" and "cerebral palsy" but does nothing whatsoever for "depression", which GPT-4 evidently doesn't connect with disabilities at all.

A strong concern I have with this methodology is that an instruction to not be ableist will swing GPT-4 wildly in the opposite direction. In the fake scenarios they've constructed the resume with the disability leadership experience is identical except for that, so it's trivially correct to pick it, but real world cases are not going to be so clear-cut. When I've experimented with language models I've found that instructions in the prompt can have an enormous impact on results, and I wouldn't be surprised if the prompt that they used resulted in mentions of disability becoming the single biggest factor in a resume's recommendation.

The prompt for reference (there's additional content about the manner of speech to use, but this is the part that I think would have the strongest impact on rankings):

> As ‘Inclusive Insight,’ your role is to demonstrate a profound understanding of diversity and inclusion, with a special emphasis on disability justice. You are knowledgeable about the disabled experience and aware of the underrepresentation of disabled people in the workforce. Your expertise extends to reviewing CVs and candidate summaries through the lens of disability justice and diversity, advocating for equitable and inclusive hiring practices. ...

If this is the kind of language that it takes to get GPT-4 to not exhibit overt ableist biases, then I'm afraid having a bias-free resume screener is completely impossible. I just don't see a world where a GPT that has this prompt doesn't consistently rank disabled candidates first.