Hacker News Reader

Comment by bberrry | Hacker News Reader

I was under the impression that Dual-n-Back[0] had real benefits. Is that not the case?

[0] http://brainworkshop.sourceforge.net/

From a meta-analysis:
" The 20 studies included here were all completed between 2008 and 2013....Sample sizes of treatment groups varied between 7 and 36 participants, and control groups between 8 and 43"
"net effect of n-back training on Gf outcome measures, about the equivalent of 3–4 points on a standardized IQ test"
ie: very small groups, tiny effect. Sounds dubious to me, like much research in the social "sciences"

Improving fluid intelligence with training on working memory:a meta-analysis http://scottbarrykaufman.com/wp-content/uploads/2014/08/au-e...

Considering the standard deviation for IQ is 15 points, I wouldn't drink an extra glass of water per day for another 3-4 IQ points, let alone play a training game for any extended amount of time. 3-4 points is statistically meaningless.

"Considering the standard deviation for IQ is 15 points"

The standard deviation for the distribution of IQ is 15 points.

The standard deviation of the errors of measurement associated with measuring IQ is 3 points.( ie: the standard error of measurement aka SEM)

I would be willing to have 3 fingers of my left hand ( not including my thumb ) amputated to improve my IQ by 1 standard deviation.

For more on Standard Error of Measurement, see http://www.csus.edu/indiv/b/brocks/Courses/EDS%20245/Handout...

What is your point? The parent made no error in his language. The inferred meaning of 'standard deviation' in this context is 'standard deviation for distribution'. The parent referred to an improvement of 3-4 IQ points, not standard deviations.

> The parent made no error in his language.

The parent claimed a 3-4 point in an individual's IQ is statistically meaningless by point to the standard deviation of IQs for an entire population. That is either (a) misguided or (b) intentionally misleading.

The standard deviation of a non-identical population has no relation to the statistical significance of a change for an individual.

Let's say the standard deviation of heights for males is 2.8 inches; that is what some of the internet claims. Let's use two standard deviations as statistically significant. That means, if someone woke up one day and was 6'3" instead of 5'10", that was not a "significant change" because they only changed in height by 5 inches.

Standard deviation for a non-identical population is completely unrelated to the significance of changes for an individual.

I believe the point is that standard dev for distribution is complete meaningless when talking about measuring a person's iq and increase thereof.

Out of respect for people who don't like being surprised, that link is a PDF.

That link is a PDF.

One standard deviation is fairly significant, though.

I think a lot of people would do mind training once per day for a possible 3-4 IQ increase, honestly. I would.

It's also possible that it only results in a fairly small general IQ increase, but a larger increase in some specific facet of recall or cognition.

It's also possible that it results in an IQ increase only for a specific subpopulation that you may or may not be a part of, e.g. the linked metanalysis says that "international studies tend to find more transfer than U.S. studies" and it does not seem as though the effect for the U.S. studies was significantly different from zero.

How can you claim a quarter of a std dev is statistically meaningless? Seems to me that's dependent on the confidence intervals involved not the standard deviation.

If you're of average IQ, an improvement of 0.2 standard deviations is about 8 percentiles of the population - the difference between being 15th in the class and being 12th or 13th. May not sound like much, but if we assume that translates directly into income (i.e. people of 10th percentile intelligence earn 10th percentile salaries) then that's the difference between a salary of $52k and $63k.

At the high end it's even more pronounced as the tail thins. Going from +2SD to +2.2SD is the difference between 98th percentile and 99th.

Bearing in mind how loosely IQ and income are correlated IRL, and the opportunity cost of brain training vs devoting the equivalent time to job-specific learning, I'd say it's far from clear a 0.2 SD boost in adult IQ should lead to any change in income at all.

If it was 1 SD, the difference in ability would be rather more difficult not to notice.

0.2 standard deviations is a very large effect.

In the metanalysis, Hedges' g is 0.24. Hedges' g is a less biased measure of Cohen's d, and Cohen d between 0.2 and 0.3 as a "small" effect (https://en.wikipedia.org/wiki/Effect_size#.22Small.22.2C_.22...). Of course there are plenty of caveats to assigning importance to effects of different magnitudes without regard to what is being measured, but this isn't a very large effect.

Here are a few ways you might be interested in measuring effect size:

- if the environment changes such that this effect becomes operative, what sort of change should I expect in terms of z-scores? ("how does the effectiveness of damping sound with crumpled paper compare to the effectiveness of mining with dynamite?")

- if I aim to change a quantity using this effect, what sort of change can I expect relative to the existing known ways of changing the quantity in question? ("How many laborers could I replace with one bundle of dynamite while ending up with the same size of hole?")

- if I see a change of so many standard deviations in some intangible variable, (a) what sort of effect will I see further down the pipeline in the variables that I really care about, or (b) is that amount subjectively worth the effort? ("If I have $600,000, can I make a bigger hole by hiring and outfitting diggers, or by buying and detonating dynamite?")

You're insisting on the first of those questions and only the first. The comment I responded to is explicitly phrased in terms of the third question, and 3-4 IQ points is quite significant in terms of tangible knock-on effects. It's also worth noting that an intervention yielding 3-4 IQ points is staggeringly large in terms of question #2, losing out to curing malnutrition but beating basically everything else. It is so large as to seriously damage the credibility of the result, given what we already know about efforts to raise IQ.

Gwern's done extensive research into this, and his meta-analysis has shown there's "a net gain (medium-sized) on the post-training IQ tests"[0, 1].

0: http://www.gwern.net/DNB%20meta-analysis

1: http://www.gwern.net/DNB%20FAQ

Literally the next paragraph:

"The size of this increase on IQ test score correlates highly with the methodological concern of whether a study used active or passive control groups. This indicates that the medium effect size is due to methodological problems and that n-back training does not increase subjects’ underlying fluid intelligence but the gains are due to the motivational effect of passive control groups (who did not train on anything) not trying as hard as the n-back-trained experimental groups on the post-tests.

The remaining studies using active control groups find a small positive effect (but this may be due to matrix-test-specific training, undetected publication bias, smaller motivational effects, etc.)"

http://www.gwern.net/DNB%20meta-analysis#analysis

This is really great! I'll have to look into this when I have a little while. Thanks for passing it on. Just on a glance, I disagree that it really shows much of an increase though. Just glancing at the forest plots of those effect sizes tells me that this isn't a very strong effect at all (if it's there).

And there is also the so called 'file drawer' effect. I was at one of the big cog psych conferences a few years ago when a colleague was asking around -'Do you have any failed to replicates for WM training?' Everyone was so excited with the original Jaeggi 2008 paper, went out and tried it, and had a tough time replicating what turned out to be a severely flawed study.

Not to say that I'm not open to the idea. I'd love for it to be true, I just think large-effect-size effects are not often mired in the controversy that this one is. They're hard to find.

Why is this downvoted? I was under the impression that Gwern did solid resarch and was well respected in the HN community.

Because JumpCrisscross's post [0] implies that w1ntermute cherry picked a component of the Gwern meta-analysis to prove his point while the next paragraph refuted it.

[0] : https://news.ycombinator.com/item?id=10846443

So it definitely transfers 'near'. You'll get better at the dual-n-back test. And possibly other visio-spatial working memory tests. But the question of far transfer- will it make you smarter- is probably not. Here is a link to a 2015 meta-analysis that looks at the question of working memory training (including dual-n-backs) transferring to other working memory tasks:

https://www.dropbox.com/s/084fvteji1tyz8t/2015-schwaighofer....

Melby-Lervag & Hulme, 2013, is also pretty damning.

That's exactly, if my understanding of changes happening to brain while performing any activity in general is correct, what should happen - you get better at doing that activity. So, unless your work contains some kind of dual-n-back activity, you are not getting better at it. You get better at work by doing work, not by playing brain game.

I am really happy they got fined for this deception.

Reply on news.ycombinator.com