Hacker News Reader

Comment by rayiner | Hacker News Reader

Look at the exam and scoring rubric: https://kaisoapbox.com/projects/faa_biographical_assessment/

Look at questions 29 and 33. The first (about whether negative feedback bothers you) is a plausible question but the grading is completely nonsensical. The second question, about art/dance classes you took in college, is nonsensical both as to the question and the answers. These seem obviously designed to be gamed with secret information.

This was used as a mandatory screen for several years. The FAA didn’t fix it, Congress found out and banned it. How many people at the FAA saw this and green lighted it?

View on HN

I found the question and answer weights as well as some primary source documents on the methodology in the court file.

My interpretation is that this was not "obviously designed to be gamed with secret information", it was just bad methodology. They had a goal of screening out 70% of applicants, but the remaining 30% of applicants needed to have a demographic balance that would not constitute a disparate impact, and they had to do this in a legally defensible way. Working backwards from that goal, they took biographical data that they had collected in the 80s and 90s and constructed a test program that, they believed, would give them the result they wanted. So if the answer weights are logically nonsensical, it's not because they were building in a secret password, it's because that's what happens when you build a model that's overfit on a small number of datapoints.

https://storage.courtlistener.com/recap/gov.uscourts.dcd.182...

https://www.courtlistener.com/docket/4542755/139/24/brigida-...

Wow. I thought I remembered something like that but I thought it sounded too extreme and wouldn't be easily supportable, so I left it out of my other comment. So this test:

1) was designed to statistically select for members of favored identity groups and against members of disfavored identity groups, and not in any way to measure ATC job aptitude, resulting in highly-scored questions like "The high school subject in which I received my lowest grades was" where the only correct answer was "Science", and failing the test disqualified you permanently

and

2) current FAA employees distributed the exact answer key to outside racial identity organizations to give to their members

What were they fitting for?

What is your source for how the biographical assessment was graded? If you read the tracingwoodgrains blog post, he is saying that Shelton Snow, the guy from the association of black employees left voicemails telling people what the correct answers to the biographical assessment questions were. But if you read my comment above, this person seems to have been pretending to have insider knowledge and access - the secret buzzwords he sent around seem to just be a photocopy from a generic resume writing book. I don't think anything that Snow said about the application process can be taken at face value.

If you have another source for how the assessment was actually graded, I'd love to see it, but as far as I can tell, these claims are coming from a guy who seemed to just be making stuff up.

The scoring rubric is in the litigation documents (in the Google drive linked in the website). ECF 139-26 has one copy.

Reply on news.ycombinator.com