Hacker News Reader

Comment by mgirdley | Hacker News Reader

Codeup CEO here. We actually did some brief testing with classical IQ tests and saw two problems:

1) IQ tests are really long and expensive, so difficult to implement

2) They didn't correlate with performance as well as algebra.

I've read your answer about your methodology of gauging "programming aptitude" and I'm worried it might be susceptible to your judges' biases - it being purely subjective. I'll put more faith in your point number 2 if you come up with a more objective measure.

Agreed that an objective measurement would be ideal and this is second best.

However, we've had a hard time creating or locating an objective measurement of general programming skill. As demonstrated by the industry's non-use of standardized tests for programming, I don't think anyone else has either.

If you have a lead on something, would love to see it.

Trying to create an accurate measurement of programming skill is far more difficult than looking for correlation on tests compared with some other subjective measure.

This misguided attempt at instrumenting "skill" could prevent highly capable individuals from entering the field who don't have a solid math foundation, for societal or other reasons.

OTOH, identifying a decent mathematical thinking background as a predictor of CS potential is a pretty big win at a societal level. Teaching math is relatively cheap; you don't even have to buy computers...

I remember reading about a similar but different test that had good results. I can't find the source so this is just from what I remember. Before starting a course the students were shown short snippets of code and asked simple questions about it. They found that consistent answers were a predictor of who would do well. The theory was that people who are able to construct a mental model for the problems ended up being good at programming tasks, even if the models they had made didn't match what code actually does. Have you heard of or tried this approach?

I think you're remembering some papers from Dehnadi and Bornat, starting with http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf

That looks like the paper I was thinking of. Thanks for including te link.

http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-...

How do you know they didn't correlate with performance as well as algebra if they were too difficult to implement?

Because they tried them, then decided to not continue trying them because they didn't correlate as well and were difficult and expensive to implement.

There main test has 60 datapoints... how many tries do you think they made exactly before deciding they didn't correlate as well? Yes, IQ tests are expensive, but if you buy one test, the marginal cost of giving to an extra student is basically 0. I suspect it wasn't tested at all.

A real IQ test takes at least 2 hours of 1-on-1 time with a psycholigist to administer, so in the event that someone actually wants to do real iq testing the marginal cost is quite high.

That's what psychometricians want you too believe. There is no scientific reason it needs to be administered by a human or be so long.

The ASVAB, GRE, LSAT, MCAT, and SAT all have correlations with psychologist administered IQ tests of around 0.9. WORDSUM, which is a ten word vocabulary test has a correlation of 0.71[1].

[1]Every time I use the WORDSUM variable from the GSS people will complain that a score on a 10-question vocabulary test is not a good measure of intelligence. The reality is that “good” is too imprecise a term. The correlation between adult IQ and WORDSUM = 0.71. The source for this number is a 1980 paper, The Enduring Effects of Education on Verbal Skills. I’ve reproduced the relevant table…

The Enduring Effect of Education on Verbal Skills

http://www.jstor.org/discover/10.2307/2112492?uid=3737800&ui...

Are they really that hard I have done iq tests administered by professionals I can't recall it taking that long.

Reply on news.ycombinator.com