Camels and humps: a retraction [pdf]
eis.mdx.ac.uk
eis.mdx.ac.uk
We generally approach science research as separate from the people making it (except for questions of reputation). There is a Scientific Method and if one follows it, Science results.
Of course, hundreds of small decisions are made by scientists all the time, each potentially introducing a bias (I find the catalog of cognitive biases alone impressive). Even the decision to pursue a particular hypothesis betrays a bias (albeit unavoidable if we want to rationally make use of past learning and accomplish anything).
But I hadn't considered mental health as a key factor--such things are quite rarely discussed (nor do they have a standard reporting mechanism in scientific papers). Apart from general stigma around mental health, I suspect it's also because it undermines the image if scientist as dispassionate seeker of truth. How much more interesting, nuanced and complex would our results be if that changed?
One interpretation of the original finding that everyone who passes the test passes the course, but not everyone who fails the test fails the course, is that some people who did not have a consistent mental model before the course develop one during it (and nobody who has developed such a model loses it). That should show up in a longitudinal test: you would expect that as people develop a model, they switch from the failing to passing groups.
This would be consistent with the sequential learning model. If you did a basic class, and didn't have a model by the end of it, then when you do a more advanced class, you have even less of a chance of learning anything from it, and less of a chance of developing a model during it.
I wonder if a way of structuring a course would be to start with a sort of mental model bootcamp, where the teaching was aimed specifically at developing a model, and where nobody would progress to the rest of the course without doing so. That way, students who take longer to get ready are not confronted with later stages of the course that they will not be able to make use of, and have a progressively greater share of the teaching resources in the first stage to help them do so.
Obviously http://blog.codinghorror.com/separating-programming-sheep-fr...
> My physician put me on the then-standard treatment for depression, an SSRI.
2014 is much better. http://www.nice.org.uk/guidance/CG90
> [...] asked some statistician colleagues if they could help us recover more information from his data.
It's a shame more organisations don't have access to statistician helpers to ensure that they are being accurate and honest when seeking, interpreting, and presenting data. Perhaps this is something else that is a result of the dominance of Excel - people have collections of numbers and you can pummel them into a spreadsheet and produce some nice charts and graphs but that leads to people over-interpreting the data.
> After a lot of work, the answers were, by and large, that we couldn’t see any such differences in our data.
This is surprising to me. I remember reading the blogs around the time and it seemed like a sensible claim. I can't remember anyone digging into the data and pointing out flaws. Did they?
I think I believed it because I feel "unteachable".
EDIT: I freaking love this paper because of its discussion of a mistake made during a phase of mental ill health, and the recovery journey afterwards.
2014 looks pretty similar.
1.5.1.2 For people with moderate or severe depression, provide a combination of antidepressant medication and a high-intensity psychological intervention (CBT or IPT).
So, for moderate or severe depression, the standard initial treatment is an SSRI and therapy.
For less severe depression, though, the guidance is to start with non-pharmaceutical options, and only move to drugs if those don't work.
There's now a recognition of "subthreshold depressive symptoms" - which are troubling and unpleasant but which either would have been missed in the past or would have been treated solely with medication.
Other stuff is much more important now. "A wide range of biological, psychological and social factors, which are not captured well by current diagnostic systems, have a significant impact on the course of depression and the response to treatment."
We're using DSM IV, not ICD10, which "* also makes it less likely that a diagnosis of depression will be based solely on symptom counting.*"
To get the therapy in the UK the person would self-refer to an IAPT (improved access to psychological therapy) style course. That would carry some kind of assessment of need, and the person would thus have another check (the first would be the GP) to see if they need specialist secondary care.
The important stuff here for the OP is much more concentration on therapy not just medication; and much more concentration on how the person is coping with life not just counting symptoms. Of course, some places do this much better than others.
I'm not sure that was possible. I haven't re-read the original 2006 paper, but it sounds like the claims in the 2006 paper may simply have been false:
> I did a number of very silly things whilst on the SSRI and some more in the immediate aftermath, amongst them writing “The camel has two humps”. I’m fairly sure that I believed, at the time, that there were people who couldn’t learn to program and that Dehnadi had proved it. Perhaps I wanted to believe it because it would explain why I’d so often failed to teach them. The paper doesn’t exactly make that claim, but it comes pretty close. It was an absurd claim because I didn’t have the extraordinary evidence needed to support it. I no longer believe it’s true. I also claimed, in an email to PPIG, that Dehnadi had discovered a “100% accurate” aptitude test (that claim is quoted in (Caspersen et al., 2007)). It’s notable evidence of my level of derangement: it was a palpably false claim, as Dehnadi’s data at the time showed.
Edit: Closest thing I could find: http://vanisoft.pl/~lopuszanski/public/canihascs/
var C = [
[[1],[2],[3],[4],[5],[6],[7],[8],[9],[10],[11]],
[[1 , 2],[3 , 4],[5 , 6],[7 , 8],[9 , 10 , 11]],
[[1 , 2 , 3 , 4],[5 , 6 , 7 , 8],[9 , 10 , 11]],
[[1 , 2 , 3 , 4 , 5 , 6 , 7 , 8],[9 , 10 , 11]]
];
The test doesn't alert() you to what model you are. You have to set a breakpoint and poke around the code. When I took it I was consistent across all 12 questions with model 2.
You can see the what model each answer corresponds to here: http://vanisoft.pl/~lopuszanski/public/canihascs/questions.j...
The question:
a = 10;
b = 20;
a=b;
If you really want to test non-programmers native skill for working with computer, you should at least briefly explain how the computer will read this statements. i.e. the computer interprets the statements sequentially, and reads the '=' symbol as 'becomes', not as 'equals'.
Without any previous understanding of what computer programming is, what it does, or how it works, and relying solely on elementary mathematical learnings, is there a particular reason that one would assume that the first two lines are directives and the third line is what we are being asked to validate? I am too far down the rabbit hole to intuitively know if that is the case, can someone else suggest whether this is a plausible conjecture?
Men are mortal
Socrates is a man
Socrates is not mortal
x = x + 1
If the situations are analogous, my guess would be that you would assume that each of these statements is an assertion, and that at least one of them must be false. Intuitively, I'd guess that it's the last one that people would assume would be false, because as you're reading from top to bottom, you've already "accepted" the first two.
If you take it to mean "The cardinality of an infinite set", "X + 1" to mean "The set X with one more element added to it", and "X = Y" to mean "X and Y have the same cardinality", then "X = X + 1" is entirely true.
Mathematics, like programming, is ultimately founded on definitions.
It seems to be much easier for people to attach the new construct += to to a new idea. I don't bring in the x=x+1 form until they have had plenty of use assigning other expressions to x. Kids don't seem to have any problem with x=y+1. They just need a little time for that idea to set properly before they start mixing things up.
A much less narrow assumption is required to reach answers of "false". Think of each example as a system of simultaneous equations.
Since this test was seemingly designed relying on the idea of destructive updates, none of the given examples have satisfying assignments. But of course standout easy-pick answers of "no solution" would ruin the test. I'd really like to see a similar study of psychologies that took into account different programming bases. Perhaps such a test would even be a good way to sort students into separate intro classes that used a language suited to their preexisting mental model.
LET A = 10
LET B = 20
LET A = B
10 → A
20 → B
B → A
put 10 into a
put 20 into b
put b into a
(* multiple bindings, so the inner x shadows the
* outer x---indeed, this code would give you a
* warning about the unused outer x *)
let x = 5 in
let x = 6 in x
(* y is a mutable reference cell being modified *)
let y = ref 5 in
y := 6; y
-- this defines x to be one more than itself---which
-- causes an infinite loop of additions when x is used
let x = x + 1 in x
-- a different assignment operator is used in a context
-- in which side-effects might occur, and can be
-- interspersed with non-side-effectful bindings:
do { x <- readInt -- does IO
; let y = 1 -- doesn't do IO
; return (x + y)
}But that's not the question that was asked. The question is "what are the final values of a and b?". Someone adopting your interpretation would say "a=10, b=20". Otherwise, why else would you be claiming "a does not equal b"?
> "If you really want to test non-programmers native skill for working with computer, you should at least briefly explain how the computer will read this statements. i.e. the computer interprets the statements sequentially, and reads the '=' symbol as 'becomes', not as 'equals'."
That would somewhat defeat the point of the test, which is to gauge what mental models (if any) people have before they've been told anything about programming.
"His test is not a very good predictor: most of those who appear to use a model in the pre-course test pass the end-of-course exam but so do many of those who do not."
How many pass, how many do not? That is the information needed to determine a predictive test.
Reading between the lines, I would guess that he got in trouble because his original article wasn't politically correct. One line in the retraction reads "We hadn’t shown that nature trumps nurture. Just a phenomenon and a prediction." I would guess he came under political pressure, and felt pressured to write this article in order to avoid further problems.
He has no real criticism of his original article. What he says about it is incoherent, not merely hard to read, as I point out in my comment.
And he makes reference to nature vs nurture, which is completely irrelevant to the actual issue, and only makes sense in the context of a political apology.
http://retractionwatch.com/2014/07/18/the-camel-doesnt-have-...
... comes the following quote from the author of the camel-humps paper:
"I presented our latest results about 18 months ago at a PPIG workshop/conference in the UK. I felt it was helpful, since the claims I made had provoked hostility to the work, to retract those claims verbally. It had a dramatic effect, to the good. But I found (how I know is the confidential bit) that there are people who didn’t hear that retraction, and who are still hostile; and that hostility is doing harm. So I decided to retract more publicly.
"Interestingly, one person who I would have counted previously as hostile heard (indirectly) of the verbal retraction, and this summer was more than supportive. Research inspired by our work is going forward. So the retraction was worthwhile."
So, confirmed: there was an angry backlash, and this retraction is an attempt to calm people down.