Comment by eitland | Hacker News Reader

Comment by eitland | Hacker News Reader

Parent Full thread

It is good to debunk bad science.

Just be careful not to repeat the same mistakes in reverse, throwing out something that is clearly helpful because the original experiment cannot be trusted.

I'll post a very simple thought experiment to prove it for one of the cases:

> No good evidence that tailoring teaching to students’ preferred learning styles has any effect on objective measures of attainment. There are dozens of these inventories, and really you’d have to look at each. (I won’t.)

Now, do as in math and consider what happens when we approach zero:

- what happens if a student cannot hear?

- what happens if a student cannot see?

or infinity:

- can you think of a learning style that works better or worse for people who are strongly affected by ADHD?

- same but for social anxiety?

So, just by thinking about it for a few moments we can debunk the debunking: Clearly there exists whole subsets of the population (blind, deaf) where one or more learning styles are impossible, meaning that the rest of the learning styles must be better for those.

What can we learn from this?

Maybe to be sceptical, even towards the sceptics?

That's a good approach, thanks for writing it as a reminder. This takes me back to math and physics lessons, where we tested a given function/formula at extremes and one or more interesting points in the middle, just to get a good feel for its behavior.

Continuing your mental framework, the problem with interpreting such research is that the results we get are in the "probe some points to get a feel" category. Consider a single learning style, and let's plot its known effectiveness, as a function of "learning capability", where "learning capability" is a measure that places students with relevant disabilities on the left, and students with relevant gifts on the right. Assume, for the sake of the exercise, that this metric is sound. We get something like:

  EFFECTIVENESS
     ^
     |                               oo
     |           x   x x
     | 
     |oo                       
     +--------------------------------->
              LEARNING CAPACITY

Where 'o' are obvious values derived from the thought experiments like you did ("what happens if a student cannot hear?", etc.), and 'x' are points from research of the method over a population of a given learning capacity. Looking at 'x' points alone would make you think the method doesn't improve anything for anyone. Looking at 'o' points alone would make you conclude it obviously is effective. But the points I depicted above are not enough to tell you whether the function looks like this:

  EFFECTIVENESS
     ^                              ***
     |                           *** 
     |   ************************
     | **
     |**                       
     +--------------------------------->
              LEARNING CAPACITY

Or like this:

  EFFECTIVENESS
     ^                           *******
     |                    *******
     |             *******
     |     ********
     |****                       
     +--------------------------------->
              LEARNING CAPACITY

Or something else entirely. This is something to keep in mind too: does the research data you saw, whether from single or multiple studies, sampled enough of the search space to give you some confidence in the conclusions?

Thanks! Good points!

That's why you use multiple approaches if you could. No real reason to limit yourself to only one preferred modality unless you're forced to.

In case there was any reason to doubt: I agree.

Isn’t the problem you’re raising here simply that the studies tend to look at populations rather than individuals or even sub-groups? It’s a neat way to demonstrate the problem. The challenge is that as long as we treat the RCT as the primary means by which we build knowledge, it will remain difficult to study sub-groups because it’s so hard to obtain a sufficient sample.

There's the old saying "absence of evidence is not evidence of absence".

Reply on news.ycombinator.com