Hacker News Reader

Comment by Paul_S | Hacker News Reader

Does a "simple algebra test" correlate with IQ? Does "programming aptitude" (whatever the hell that is meant to be) correlate with IQ? The headline would then follow.

I know this is a socially unacceptable opinion but I think those correlations exist.

View on HN

Codeup CEO here. We actually did some brief testing with classical IQ tests and saw two problems:

1) IQ tests are really long and expensive, so difficult to implement

2) They didn't correlate with performance as well as algebra.

I've read your answer about your methodology of gauging "programming aptitude" and I'm worried it might be susceptible to your judges' biases - it being purely subjective. I'll put more faith in your point number 2 if you come up with a more objective measure.

Agreed that an objective measurement would be ideal and this is second best.

However, we've had a hard time creating or locating an objective measurement of general programming skill. As demonstrated by the industry's non-use of standardized tests for programming, I don't think anyone else has either.

If you have a lead on something, would love to see it.

Trying to create an accurate measurement of programming skill is far more difficult than looking for correlation on tests compared with some other subjective measure.

This misguided attempt at instrumenting "skill" could prevent highly capable individuals from entering the field who don't have a solid math foundation, for societal or other reasons.

OTOH, identifying a decent mathematical thinking background as a predictor of CS potential is a pretty big win at a societal level. Teaching math is relatively cheap; you don't even have to buy computers...

I remember reading about a similar but different test that had good results. I can't find the source so this is just from what I remember. Before starting a course the students were shown short snippets of code and asked simple questions about it. They found that consistent answers were a predictor of who would do well. The theory was that people who are able to construct a mental model for the problems ended up being good at programming tasks, even if the models they had made didn't match what code actually does. Have you heard of or tried this approach?

I think you're remembering some papers from Dehnadi and Bornat, starting with http://www.eis.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf

That looks like the paper I was thinking of. Thanks for including te link.

http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-...

How do you know they didn't correlate with performance as well as algebra if they were too difficult to implement?

Because they tried them, then decided to not continue trying them because they didn't correlate as well and were difficult and expensive to implement.

There main test has 60 datapoints... how many tries do you think they made exactly before deciding they didn't correlate as well? Yes, IQ tests are expensive, but if you buy one test, the marginal cost of giving to an extra student is basically 0. I suspect it wasn't tested at all.

A real IQ test takes at least 2 hours of 1-on-1 time with a psycholigist to administer, so in the event that someone actually wants to do real iq testing the marginal cost is quite high.

The ASVAB, GRE, LSAT, MCAT, and SAT all have correlations with psychologist administered IQ tests of around 0.9. WORDSUM, which is a ten word vocabulary test has a correlation of 0.71[1].

[1]Every time I use the WORDSUM variable from the GSS people will complain that a score on a 10-question vocabulary test is not a good measure of intelligence. The reality is that “good” is too imprecise a term. The correlation between adult IQ and WORDSUM = 0.71. The source for this number is a 1980 paper, The Enduring Effects of Education on Verbal Skills. I’ve reproduced the relevant table…

The Enduring Effect of Education on Verbal Skills

http://www.jstor.org/discover/10.2307/2112492?uid=3737800&ui...

That's what psychometricians want you too believe. There is no scientific reason it needs to be administered by a human or be so long.

Are they really that hard I have done iq tests administered by professionals I can't recall it taking that long.

Not quite. X can correlate with Y, and Y can correlate with Z without X and Z having a correlation. As a simple toy example let X and Z be independently taken from the standard uniform distribution, and let Y = X + Z.

fwiw, correlation actually is transitive if strong enough. This is obvious if corr(x,y) = 1 and corr(y,z) = 1, then corr(x,z) = 1. More generally, you can get this constraint by looking at the minimal value a for which the matrix [[1, corr(x,y), a], [corr(x,y), 1, corr(y, z)], [a, corr(y,z), 1]] remains positive semidefinite to find the minimal correlation of x and z given that of (x,y) and (y,z). I mention this because in this case the correlations may be strong enough so that the minimal a is positive, and because it's important to understand exactly when our intuition about transitivity of correlation breaks down.

For example, when x,y and y,z are perfectly correlated, we can see that corr(x,z) must be greater than 0.9 from: http://www.wolframalpha.com/input/?i=%5B%5B1%2C1%2C0.9%5D%2C...

A lot of things "correlate with IQ" to varying degrees. Education level, length of sleep, size of vocabulary, etc. I don't understand why the relationships you suggested would be socially unacceptable.

But just because those two things correlate with IQ (which is probably true, so let's just grant them as true even without evidence for the sake of argument), that does not imply that they are sufficiently strongly correlated with each other to be interesting for hiring purposes.

To see my point, consider that being red is correlated with being purple, and being blue is correlated with being purple, but being red is not correlated with being blue. (at least not without evidence).

I don't think the existence of those correlations depends on what you think. Har har. Suggesting a correlation is a statistical hypothesis that is testable but not really debatable.

I don't think such a hypothesis is awkward but I doubt high IQ as an excellent predictor for an individuals performance in tasks that are not IQ tests. As a layman when it comes to psychology I think IQ measures some things but the way the brain works, no real world task a person does is exactly like those IQ tests.

A brain is not a CPU and an IQ is not its clock frequency.

Aside from being wrong and indeed often disparaged, this still common attitude has the unfortunate effect of helping to exclude those who can make an excellent contribution to a programming team but who approach programming from a different point of view - for example, I have a friend who's working quite competently in the field and has general block with respect to math but came to programming through music and graphic design.

>"programming aptitude" (whatever the hell that is meant to be)

What's not clear about the term?

We as an industry have no objective measure for what is and is not a good program. Therefore, we have no objective measure for who is good at programming.

If you ask me, a good program is one that is documented accurately and thoroughly. So a good programmer is one who cares enough to document well, and has enough skill to do it with precision, speed, and with good judgement.

Of course, being a good scientist or mathematician requires the same skills: know who you are talking to, what they already understand, and why your work matters to them. And be obsessive about doing it the right way.

There is virtually no evidence that program documentation leads to "better" programs. In fact, there are many people that will maintain that the need for code documentation is evidence of a "bad" program.

Regardless of whether you subscribe to this theory or the opposite, there is no "objective" way to differentiate between these 2 spectrums.

No evidence? No evidence for a _definition_?

I am telling you what a good definition of a good program is. Of course there is no evidence that my definition is the same as your (unstated) definition. You can disagree, but this has nothing to do with evidence.

Reply on news.ycombinator.com