Hacker News Reader

Comment by comeonbro | Hacker News Reader

I'm sure this is multifactorial and the other factors are probably as or more important, but especially if you are familiar with the very egregious specifics of the FAA diversity hiring scandal (https://www.tracingwoodgrains.com/p/the-faas-hiring-scandal-...), "is unsupportable" here is a strong claim that is playing the classic and well-worn role that "there's no evidence" often does in bad journalism: https://www.astralcodexten.com/p/the-phrase-no-evidence-is-a...

View on HN

I dug into one of the claims in the blog, and as far as I can tell, he's completely misinterpreting the evidence. The blog says:

> An FAA employee and then-president of the NBCFAE's Washington Suburban chapter, provided NBCFAE members with "buzz words" in January 2014 that would automatically push their resumes to the tops of HR files.

It's true that this person said that in the email, but if you actually look at the list of buzzwords, it's clear that this person was bullshitting and inflating his own importance (or maybe just fundamentally misunderstanding something). The list is on the last page of this document: https://drive.google.com/drive/folders/17Vi9dDtZvbwHDafrygRG... It looks like it's a page that was photocopied from a book about how to write a resume. It's a list of dozens of incredibly generic verbs like "manage", "analyze", "administer", "make", "improve", "design", etc. pretty much any resume will have at least some of these verbs. There's just no way you could build a system that would use these verbs to secretly screen in resumes of people who are in the know because everyone uses these verbs. A far more plausible explanation is that this guy was trying to make himself sound more powerful than eh really was.

The email this person sent is awful, but it seems to be one person's very misguided attempt to push an agenda, not some sort of secret plot by FAA.

Look at the exam and scoring rubric: https://kaisoapbox.com/projects/faa_biographical_assessment/

Look at questions 29 and 33. The first (about whether negative feedback bothers you) is a plausible question but the grading is completely nonsensical. The second question, about art/dance classes you took in college, is nonsensical both as to the question and the answers. These seem obviously designed to be gamed with secret information.

This was used as a mandatory screen for several years. The FAA didn’t fix it, Congress found out and banned it. How many people at the FAA saw this and green lighted it?

What is your source for how the biographical assessment was graded? If you read the tracingwoodgrains blog post, he is saying that Shelton Snow, the guy from the association of black employees left voicemails telling people what the correct answers to the biographical assessment questions were. But if you read my comment above, this person seems to have been pretending to have insider knowledge and access - the secret buzzwords he sent around seem to just be a photocopy from a generic resume writing book. I don't think anything that Snow said about the application process can be taken at face value.

If you have another source for how the assessment was actually graded, I'd love to see it, but as far as I can tell, these claims are coming from a guy who seemed to just be making stuff up.

The scoring rubric is in the litigation documents (in the Google drive linked in the website). ECF 139-26 has one copy.

I found the question and answer weights as well as some primary source documents on the methodology in the court file.

My interpretation is that this was not "obviously designed to be gamed with secret information", it was just bad methodology. They had a goal of screening out 70% of applicants, but the remaining 30% of applicants needed to have a demographic balance that would not constitute a disparate impact, and they had to do this in a legally defensible way. Working backwards from that goal, they took biographical data that they had collected in the 80s and 90s and constructed a test program that, they believed, would give them the result they wanted. So if the answer weights are logically nonsensical, it's not because they were building in a secret password, it's because that's what happens when you build a model that's overfit on a small number of datapoints.

https://storage.courtlistener.com/recap/gov.uscourts.dcd.182...

https://www.courtlistener.com/docket/4542755/139/24/brigida-...

Wow. I thought I remembered something like that but I thought it sounded too extreme and wouldn't be easily supportable, so I left it out of my other comment. So this test:

1) was designed to statistically select for members of favored identity groups and against members of disfavored identity groups, and not in any way to measure ATC job aptitude, resulting in highly-scored questions like "The high school subject in which I received my lowest grades was" where the only correct answer was "Science", and failing the test disqualified you permanently

and

2) current FAA employees distributed the exact answer key to outside racial identity organizations to give to their members

What were they fitting for?

You are conflating multiple different instances of current FAA employees providing outside racial identity groups with both hard and soft answer keys, the buzzword filter was just the first stage.

Later and probably most egregiously, there was a completely nonsensical and arbitrary biographical questionnaire which was scored like:

https://storage.courtlistener.com/recap/gov.uscourts.dcd.182...

    15. The high school subject in which I received my lowest grades was:
    A. SCIENCE (+15)
    B. MATH (0)
    C. ENGLISH (0)
    D. HISTORY/SOCIAL SCIENCES (0)
    E. PHYSICAL EDUCATION (0)

    16. Of the following, the college subject in which I received my lowest grades was:
    A. SCIENCE (0)
    B. MATH (0)
    C. ENGLISH (0)
    D. HISTORY/POLITICAL SCIENCE (+15)
    E. DID NOT ATTEND COLLEGE (0)

    29. My peers would probably say that having someone criticize my performance (i.e. point out a mistake) bothers me:
    A. MUCH LESS THAN MOST (+8)
    B. SOMEWHAT LESS THAN MOST (+4)
    C. ABOUT THE SAME AS MOST (+8)
    D. SOMEWHAT MORE THAN MOST (0)
    E. MUCH MORE THAN MOST (+10)

where current FAA employees, again, distributed the exact answer key to outside racial identity groups to give to their members.

If the buzzwords that Snow provided turned out to just be something he pulled from a resume book rather than insider information, why should we believe that the answer keys were any more accurate?

Reply on news.ycombinator.com