Results from Khan Academy experiment | Hacker News Reader

Results from Khan Academy experiment

blendmylearning.com

39 pointsmikeleeorg13 comments

In the comments, the authors give the sample size as about 20 in each group. Here's a more sensible graph (error bars to create a 95% CI on the estimated group means):

https://skitch.com/johnjosephhorton/fih62/quartz-2

Obviously, there are some statistical power issues. It's unfortunate that they didn't get someone to help them with a power analysis up front---given how many very tech savvy people are interested in Kahn Academy and want to see it succeed, I think they could have gotten this sort of assistance pretty easily.

Thanks for doing that, I found myself wishing they had put error bars on when I saw that graph but was too lazy to do it myself.

With respect to the statistical power issue, it may be they just didn't have enough students. Unlike your typical clinical trial or scientific study where you're recruiting people, I got the impression this was from an existing pool of students for a specific school. Once you go beyond this, the design gets tricky since there are practical issues to how big a class can be. Then you need more teachers, and you need to normalize for per-teacher variability. Not saying a study like that wouldn't be great, but it would be complex to get right.

no problem & agreed about what probably happened - my point was more that knowing they had that sample size (n=40), someone could have told them that even if Kahn academy was extraordinarily effective, there was almost no chance they'd detect statistically significant effect. If they knew that, they might have pursued a more qualitative approach rather than have people see the take-away as "no effect."

I try to give educationalists the benefit of the doubt, because they probably mean well, but at the same time, this stuff is important so I want them to be held to a higher standard.

I can't help but feel that this image they used sums up the whole project. Two (basically meaningless) numbers, presented in a 3-D bar chart. http://blendmylearning.files.wordpress.com/2011/08/resultsgr...

The only "result" that the article could claim is the unfortunate There was no significant difference between the two teaching methods.

However, we could probably regain hope for Khan Academy by pointing out that the (five-week) summer school was almost entirely worthless for 2/3rds of the students anyway:

In terms of distribution, in each class approximately one third of the students saw some significant gains (ten percent or higher gains in percentage of questions answered correctly), whereas two thirds of the students’ scores were essentially flat (less than four percent increase or decrease).

If anything, it is this that should make us eager to explore alternative methods of teaching. I am hopeful that Khan Academy or a similar online learning provider will revolutionise education, and this article is the first step along the way to discovering how to use it.

I'm no statistical whiz but I'm not sure they can even claim There was no significant difference between the two teaching methods, but rather we have not demonstrated a signifiant difference (i.e. absence of proof vs proof of absence). Despite being supportive of the Khan Academy, I can't conclude that it isn't in fact terrible compared with traditional methods on the basis of this experiment. And given that, if this article is the first step, it's the first step in the wrong direction, like starting out to optimise software based on a faulty benchmark.

I think you may have missed what I think is the most important point made in the article:

"It is also interesting to consider that students in the treatment group spent approximately half of the summer working on pre-algebra skills. Because the Khan software is individualized, it identified that most of our students had significant pre-algebra skill gaps and delivered instruction and practice problems to address these deficits. Students in the Khan/treatment group therefore spent up to 50% less time than the control group on the algebra content that the MDTP exam measured. The treatment group, however, still performed at a similar level the control group on the algebra measures."

That says to me that these students spent most of their time filling in gaps in their knowledge and still managed to keep up and even get ahead of the other class in a very short amount of time. They're basically saying flat out that these students learned as much of the advanced material in half the time as the other group and probably have better foundational knowledge.

No significant difference my ass.

I really hate it when people (mathematicians?) misuse the percentage notation...

> increased their percentage of correct answers by 5.2% over the five-week period > ... > For example, a student who started the summer knowing 60% of the correct answers in the traditional class ended the five weeks knowing 65.2% of the correct answers.

This means that the percentage of correct answers increased 5.2 percentage points, which is actually an 8.67% improvement (60% + 60% * 8.76 / 100 = 65.2%).

> which is actually an 8.67% improvement (60% + 60% * 8.76 / 100 = 65.2%).

Yes, it is 8.67%.

Illustrating the calculation is better served by:

Initial: 60

After: 65.2

Increase: 5.2

% Increase: 5.2 / 60 * 100 = 8.76

8.67

I thought the Khan Academy was about 'leave no learner behind'. That is, if you miss a concept in traditional school, you flail around the rest of the term. But in Khan Academy, you can review lessons, stop and start them, ask limitless questions in the middle because they're all taped.

SO, I would expect to see extensive data on the distribution of the scores - how long is the lower tail of the curve? Did they pick up those that would have lagged behind?

As is, they give a single percentage - improvement on an algebra test. So no meaningful data at all. Hm.

In my experience, I see Khan academy being most valuable in a blended setting when the teacher isn't very good (because Salman explains things so well), but also over a longer period of time. Khan academy encourages retaining old material in at least two important ways: it makes the old material easily accessible, and Salman's teaching method strongly encourages you to have a better understanding of the concepts behind the math (for me, that's been in proving trig identities instead of memorizing and explaining how the process of multiplying matrices is arbitrary, but it's clearly a fundamental aspect of how Salman teaches).

Warning for iPad users, since it isn't apparent from the URL: the site is at Wordpress.com and has the OnSwipe malware.

Reply on news.ycombinator.com