New York City’s AI hiring law takes effect
qz.com
qz.com
> * AI-based chatbots that ask candidates questions about their qualifications, then decide if they’ll proceed in the interview process
> * Algorithmic video platforms that have candidates answer interview questions on camera, record their replies, transcribe their responses, and analyze their vocal or facial patterns for subjective traits like “openness” or “conscientiousness”
> * Logic games that purport to identify qualities like “risk-taking” or “generosity”
Alright, which one of us monsters built any of this dystopian garbage?
(We certainly are monsters with a sense of humor: see the example of deploying Kafkaesque automated snake oil gatekeeping to people's livelihoods... to seek "conscientiousness".)
Buddy, it takes a village.
But I feel personally obligated to just note, for any HNers reading, that the two ex-McKinsey people I've worked with closely at a startup were both decent, and I'd work with them again.
Heh, maybe we can get McKinsey on our side, on this particular AI dystopia threat. Some hypothetical vendor-selection AI model (comparable to a hiring one) might take one look at certain past scandal involvements of the firm, and denylist anyone associated. But even organizations that don't care about individual fairness might still have to care about some AI pet project sabotaging the quality of their people and vendors.
I also think the argument that big companies wouldn't do it if didn't work is false. This is the sort of thing you're going to be able to create some big awesome presentation for, demonstrating that not only does it work, but it will end up saving you large amounts of money. Even if it does nothing of the sort. There's just so much room for number jukery, buzzwordery, and general shenaniganry - even if you never overtly lie.
Like most other things companies do in the hiring process (e.g. endless redundant interviewing rounds, ridiculous culture fit questions) -- or the stuff they do in their day-to-day work ("Agile") -- it doesn't need to actually "work" for it to be widely adopted.
It just needs to appear to work, and to give off a warm, buzzy feeling.
I would think that's one of the primary concerns here, is that we can pretty much guarantee that it doesn't work. Then end result would turn hiring processes into a kafka-esque hellscape where reason has no bearing and candidates are essentially selected at random.
Given how infuriating the current crop of simulated customer service agents are, I can't imagine bringing anything like this to job selection would be good for any of us, save for the leadership team who gets to tout their cost-saving plan and give themselves a bigger bonus.
Edit: ugh after reading the article I'm so disappointed that the regulations amount to some pretty weak auditing and transparency requirements. I was hoping this was an outright ban. It seems the laws have been rewritten by the cooperations. It gives candidate "right" to ask questions about their use and possibly opt out, but all that means is anyone who speaks out will be unhireable. I mean what company is going to hire anyone who starts asserting their rights up front?
That’s pretty much what it is today isn’t it. You get into a room with a hiring manager and if you happen to have kids the same age or saw the same movie growing up, you’re gonna have a huge leg up.
Can we blame someone for at least trying to make a system that’s more objective
I think the entire history of business backs you up on this.
That seems like harsh characterization. If all of the candidates are poor matches then perhaps the recruiting system isn't working properly.
I think this is inevitable, in one form or another. It will just be outsourced to some other companies which will rehash and extract whatever data, and sell services like the current background-checking companies, without even exposing any specific data to the hiring company. It's time to bit the bullet and remember that everything you say online publicly is going to remain in your permanent record, even with GDPR and stuff in place. It can't be countered by regulations, much like encryption can't be made transparent only to law enforcement by regulations. What once has been made public cannot be unmade public.
> AI-based chatbots that ask candidates questions about their qualifications, then decide if they’ll proceed
I suppose non-AI analysis of forms is not going anywhere.
> analyze their vocal or facial patterns for subjective traits like “openness” or “conscientiousness”
> Logic games that purport to identify qualities like “risk-taking” or “generosity”
I think it's just so stupid that I'd care more about my right to have disclosed the fact that a particular company does that.
On the other hand, this means that the state need to put its nose much deeper into the hiring process of a private company than it used to. It's the definition of public oversight, of course, but usually it applies to public offices.
This group will keep shrinking though.
My first job as a developer in 2006 I sat with the lead developer, had a very comfortable chat about things I had built, things I was interested in both related to programming and otherwise. Conversation turned to math and algorithms a little bit. He stepped out and came back with an offer. Whole thing took maybe 45 minutes.
I honestly feel as far as interviews have gone, it did the best of honestly sizing me up as a person.
Most companies don’t operate anywhere close to this scale and should not imitate this same hiring process.
We software engineers are used to be in high demand and somehow short supply, even now when it's taking me more than two months to find a new position. In many other industries supply noticeably exceeds demand, and people would go to great lengths and jump through really silly hoops just to get hired.
Even without the bias issues, if I were a hiring manager how do I verify that the weights "MagicHireAI SaaS Corp" are using aren't leaving swaths of talent on the table that I don't even know exist?
The amount of stuff they make you fill out that is then parsed with AI is insane.
Indeed ruined a lot. Before if you had an ok resume but we're able to talk to them in person you might be able to convince them.
Now with all those aptitude training you have to do well there.
Not to mention the applications that require you to turn on your webcam (!) while you do them. Disgusting.
That last one by the way was the same company where my wife had an in person interview. She drove there. The interview was on Zoom with somebody in another office.
She declined the offer. Wastefulness and already showing that they're not against monitoring by AI means 0 humanity.
I know I have to.
I know I don’t have to.
I don't understand what this means. Did she have to turn on her webcam to upload her cv? How does that even make sense.
Also, how is that legal? I thought requiring a photograph to apply for a position is illegal (except for stuff like acting).
> Now with all those aptitude training [...] that require you to turn on your webcam
They are referring to an automated personality or skills test to screen candidates. The webcam is to make sure the applicant is the one actually doing the test. These things are dehumanizing and snake-oil but there’s’s no question that applicant misrepresentation happens, including situations where the person interviewing is a ringer being paid by an agency or the applicant.The problem here is that companies have now gone from identity verification to this 'aptitude test' ML bullshit that's just another cover for increasingly weird arbitrary discrimination against applicants.
It doesn't benefit genuine candidates to blatantly lie. They'll get caught later in the process. But you're assuming identity thieves are acting rationally or fear getting caught. They don't, and they're playing a numbers game. If you get past an interviewer and get fired after two weeks, your salary at a US tech company can be $5k, which is the average annual salary in many developing countries.
In the US, it is illegal per the EEOC, which has barely any resources to actually enforce the law: "Similarly, employers should not ask for a photograph of an applicant. If needed for identification purposes, a photograph may be obtained after an offer of employment is made and accepted." [1]
[1] https://www.eeoc.gov/prohibited-employment-policiespractices
Asking for a photo opens an employer to claims of discrimination, but it would still have to be proven, so the government is advising against it. But it may well be that in practice, the government will not be spending resources on trying to or be able to prove discrimination, even if a photo is required.
Don't pay for this cruft, sponsor local meetups instead.
A core lesson in ML: this is difficult or impossible. See the classic example with wolves, dogs, and snow. [1]
For a resume example, suppose you don't want to be biased again black candidates, or women. Well, good luck trying to filter out resume experience, skills, schools, and interests so that they don't list anything that is statistically more likely to be found among black folks or women. It's impossible.
So now you have a hiring process that looks at... what exactly?
[1] https://www.researchgate.net/figure/A-husky-on-the-left-is-c...
The whole point of white listing variables is to only allow things you are comfortable using todiscimrinate. And yes, that likely reduces the power of the model.
Edit: to clarify, your example had the model failing to classify something correctly because the training data did not generalize across real world examples. The problem with diversity bias is that the modes are trained on past data where the labels themselves tend to have some bias. Black candidates do worse. So recreatingthese will generally produce worse outcomes for minority candidates. Now for a lot of these cases the average minority individual is more likely to do worse. Not intrinsically because of race, but still worse.
You can’t realistically expect the model to produce equal outcomes for all classes when all classes are not equal on the actual traits. But you do want to try and make it so that the model doesn’t use race indicators as a crutch that ignores actual good indicators. The common complaint is you hear about a man and a woman applying with the exact same credentials but different outcomes. Well obviously the model had to have additional information besides those credentials if it made different decisions. If this causes problems, you probably should not permit those additional pieces of information.
The problem is that variables where you can safely discriminate don't exist in the real world because everything is intertwined.
For example, if you see that somebody went to school at {Howard, Bryn Mawr, BYU} you can make some pretty solid inferences about their {race, gender, religion} even if they don't explicitly provide them.
Howard is a historically black college, which means their student body today is majority black, not 100% black.
Bryn Mawr is a womens' school and admits only women. However, gender is not always fixed going through life.
BYU is a Christian school, but also admits several non-religious people too, and people regularly become apostates to their respective religion.
Technically true of course, but >99% of the population will identify with a single gender throughout their life. The correlation is so strong that it's not even really a correlation anymore.
That doesn't mean you aren't stereotyping people, though.
If the training data shows a history of bias against Howard graduates, the model will learn to replicate.
Job applications are kind of dumb. The common use case is credit decisions. Surely, for example, you’re ok looking at credit history and income when deciding to make a loan, right? Even though it won’t be equitable outcomes across races, it is still “fair”
Other goals can be to have equal model accuracy across subgroups (e.g., models has the same precision/auroc/etc for each subgroup, even if the predictions themselves differ across subgroups) or identical distribution of statistical biases across subgroups (e.g., the model makes the same types of mistakes within each subgroup)
Like, let’s look at that for credit decisions. Black folks have a lower income distribution. So it’s like saying the model is ok if it rejects a Black applicant making 80k if it would also reject a White candidate making 100k. That is the insane bullshit you’re effectively supporting by desiring comparable error distributions.
You have to accept that equal outcomes are not just impossible but not desirable. And then you only use fields that people agree are fair to use. If Black people get disproportionately better or worse outcomes it is either because those outcomes make sense given their stats or it’s not ok because if a White dude had the same info, that you agree is fair to use, they would have passed and the underlying reason for the difference is some bullshit other input.
That might mean that the model is a dumb: you must make $100k for this loan (illustrative) which is going to mean a lot of Black folks cannot get it. Well that model is at least “fair”. Accommodations for race should not be coming from an AI model
For example you can infer the race of a candidate from other facts about them. The model could still illegally discriminate by first inferring race, then discriminating on the basis of inferred race.
The problem you describe affects blacklists that try to remove only the things you obviously don’t want to use.
i want a highly biased sample of attributes in the extreme tail end which will have all sorts of weirdness in it.
at least 2/3 of the population are unteachable for the management/engineering i want.
i don't want applicants that look anything like normal.
is that now illegal?
Which part of the article did you skip?
Start with the part where AI hiring tools favor people named "Jared."
If you’re saying humans are consciously hiring people named Jared, they shouldn’t be allowed to provide hiring feedback.
(Or the AI was used incompetently. That's always an option. Was AI the reason why US universities discriminated illegally against non-blacks and non-hispanics? I really doubt it.)
1. Gender bias
There is a widely used keyword-based gender bias measurement based on a study from 2011. The research claims that job ads with higher levels have more masculine words. This is based on a pre-defined list of masculine and feminine words [0][1]. Another Harvard study stated that women would only apply for jobs if they meet 100% of the requirements, while men would apply for 60% or less [2].
2. Racial bias
This might be more related to video interviews or based on name, age, or address in resumes. For resumes, there are tools already to anonymise resumes to ensure the focus is on experience instead of the person.
3. Disability
ADA (American Disability Act). According to this FAQ [3], "The ADA does not require employers to develop or maintain job descriptions. However, a written job description that is prepared before advertising or interviewing applicants for a job will be considered as evidence along with other relevant factors."
The list could go on, with age, sexual orientation, native speaker or not ... etc. That's why the expectations have to be clearly defined.
The law, as I understand it, will lock employers to their vendors as they will be required to keep data for auditing, which might incur more charges.
[0] https://gender-decoder.katmatfield.com/static/documents/Gauc...
[1] https://gender-decoder.katmatfield.com/about
[2] https://hbr.org/2014/08/why-women-dont-apply-for-jobs-unless...
[3] https://adata.org/faq/does-ada-require-employers-develop-wri....
https://www.govinfo.gov/content/pkg/USCODE-2010-title42/html...
https://www.ecfr.gov/current/title-29/subtitle-B/chapter-XIV...
The NY law uses the word ‘bias’ pretty much only in the phrase “bias audit”, which is defined in the law to mean a yearly audit of the software to make sure it is not discriminating as far as EEOC standards.
The point is that there isn’t a new law about bias, and it’s not an ambiguous or undefined concept. It’s the existing equal employment opportunity laws, and all employers already know about them, and most (I hope) already adhere to them. I’m not suggesting it’s either complete or perfect, I’m just pointing out what the new NY law is referring to when it uses the words “bias audit”.
I'm not sure what can be done with these cases, and I know people, of various protected out-groups, that have specifically mentioned those as reasons not to join a company.
I guess I'd have to generalize it from bio- to also apply to tech now.
I would like to see a distribution of salaries for jobs in NYC before saying ~100k
So if 20 women and 100 men apply, it would be considered unfair to accept 10 women and 20 men (as 10/20 != 20/100).
[1] https://legistar.council.nyc.gov/LegislationDetail.aspx?ID=4...