Hacker News Reader

Comment by tptacek | Hacker News Reader

If you could tell us how to apply data analysis to produce a concrete number that represents the merits and talents of a fellow human being, I would love to hear it; it would make my job a lot easier. I didn't know that was possible. Is this some kind of linear algebra-y machine learning sort of thing?

View on HN

Most likely it is possible. Take this "Alex" you frequently speak highly of. If you gave Alex a different work sample test, do you believe he'd have done poorly on it? If not, then there is some common factor (call it H) both tests are measuring.

(I can confidently state that you do believe he would have performed similarly on a different test - if he didn't, this whole idea would be fundamentally invalid.)

Now the important question is to determine, for any particular subtest, how closely a person's score correlates with the hidden factor H. This is, for example, how IQ tests are created. You can take a look at the literature surrounding them. Key starting points would be principal component analysis, clustering, hidden markov models and the like.

There is no simple recipe - every data analysis problem is different in it's own way. But there are general themes. I'm happy to discuss in more detail, feel free to write to me if you are interested.

This is a good answer.

I could tell if GP was being sarcastic so I deleted my original reply.

The softer side of the equation is that the approach of filtering candidates for businesses allows them to minimize (to some market minimum) the cost for highly skilled individuals. Projects such as this just triggered a bone I'm presently picking over.

Old-guard corporations that hire the legally-liable kind of engineer seemed to have taken a much different approach if the stories I've been told by a retired mechanical engineer who worked for Chrysler are true. A completely irrational approach. They hired people based on their potential and made them into the engineers they needed.

The copy on the announcement sounds like this game will quantify everything about the performance of a participant in the game in order to sell them to a curated list of potential employers. What about this system incentives employers to invest in the career development of these people and strengthens our collective bargaining power as the people who build this stuff?

To be more constructive I might suggest turning down the hyperbole and use fewer adjectives. Keep the pitch to employers for employers. Don't be patronizing to inexperienced developers: inspecting the assembly output of a compiler is not elite and not difficult to explain to someone given the right context and framing. I understand your game is about competition but the "winner/failure" schism it can create is a big turn off for a lot of otherwise intelligent, creative, and capable people. It doesn't have to be about being, "the best," in order to be fun and rewarding.

IME, IQ tests (like other standardized tests, SAT's, GRE's etc) are easily gamed. I did a couple practice ones and my score jumped up by 20+ points the second time. Where is the evidence that it measures some hidden factor? Seems to me that (like other standardized tests) it simply measures the subject's level of training and preparation.

It depends which test is used and there are limits in magnitude to practice effects, i.e. you can't coach anyone to a perfect score in a test with a large bank of test questions. Psychometricians are aware of these critiques. Tests like the SAT, GRE, GMAT etc., lightly disguised IQ tests all, are useful for predicting academic performance among other things.

>Effects of practice on the Wechsler Adult Intelligence Scale-IV across 3- and 6-month intervals. Estevis E1, Basso MR, Combs D.

A total of 54 participants (age M = 20.9; education M = 14.9; initial Full Scale IQ M = 111.6) were administered the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) at baseline and again either 3 or 6 months later. Scores on the Full Scale IQ, Verbal Comprehension, Working Memory, Perceptual Reasoning, Processing Speed, and General Ability Indices improved approximately 7, 5, 4, 5, 9, and 6 points, respectively, and increases were similar regardless of whether the re-examination occurred over 3- or 6-month intervals.

> Practice Effects for the Stanford–Binet Intelligence Scales The Stanford-Binet Intelligence Scales—Fifth Edition (SB5) is a widely used assessment tool for measuring intelligence (Roid, 2003). According to Roid, a key advantage of this intelligence test’s most recent revision is that it includes improved lowend items for better measurement of young children or adults having mental retardation. Sbordone, Saul, and Purisch (2007) report that the range of the SB5 was expanded to allow the assessment of very low and very high levels of cognitive ability. Roid and Barram (2004) indicate that the practice effects on the SB5 were smaller than expected. For example, the nonverbal IQ of the SB5 showed shifts of only 2 to 5 points as compared to the 4 to 13 points on the Performance IQ of the Wechsler scales (i.e., the WAIS-III and WISC-III). Roid and Barram add that the lower shift, and thus practice effect, is even more notable given that the retest period for the SB5 was 5 to 8 days versus 23 to 35 days on average for the Wechsler scales.

At an absolute minimum you should be collecting lots of statistics and using them to train a bayesian candidate filter to estimate hiring probability (the same architecture as a bayesian spam filter would work).

If you find yourself wanting something more advanced than that, random forests would be a good match here, as would clustering the candidates using DBNs or simply k-means and then using base rates.

Also, if you haven't looked at https://www.kaggle.com/competitions you really should -- they're similar to what you're doing, only for machine learning.

I was hoping the challenges would include machine learning. There is so much exciting overlap between ML and security. From detecting intrusion attempts to scraping and processing pastebins.

I think it is amazing what you guys did here: you got more points on a give me your e-mail, we are building something post, than Stripe got when announcing their company. Really speaks volume on Patio's writing.

Will there also be marketing challenges? Like: Every day patio11 clicks on one email. Your task is to write an engaging headline. Or: Here is some Google Analytics code. Make it show 50.000 visits any way you can. Very hard to cheat. (But so is getting Google security to file a bug report).

I really enjoyed the comments. Especially the ipod-hasnt-got-wifi criticism. You guys must know that what you are doing is terrible and you should feel bad.

Who knew decoding morse in some wave extracted from a corrupted image file in Cicada 3301 could also lead to a job. Will there be an ARG element to the CTF's? Like you pose as an agent from the NSA or something?

Reply on news.ycombinator.com