The 2 Sigma Problem (1984) [pdf]

Keep in mind that Benjamin Bloom attributes much of the success of tutoring to the affective learning components. That is not addressed in the meta analysis.

Bloom described how the tutor and the student achieved an emotional connection that is often difficult to achieve with a class of students.

This is so critical, because the barrier to learning for many students is emotional. It’s not that they are really trying and just don’t get it; it’s that they don’t have the capacity to care enough to engage over time. Emotional barriers to learning are super widespread — being “bored” for instance is an emotional response. Transformational learning takes place when there is an authentic emotional motivation to succeed. Human contact can support that. It’s also why stuff like ALECKS only goes so far. There is not emotional resonance, like with a human.

Thanks. My takeaway from the Nintil article is that nobody has performed a good study on the effectiveness of mastery learning vs traditional teaching. All of the studies have some fatal flaw: not randomized, small sample size, study duration too short, interval between exams too long, not providing specialized remedial content to students, or not actually requiring mastery.

I think the massive effects shown by software tutoring in the DARPA studies point to the mechanism: frequent exams and specialized remedial content. Good tutoring software continually tests students for mastery, identifies specific misconceptions, and provides specialized remedial content for each misconception. The automated software can perform this iteration for each core concept, multiple times per hour. Students frequently get feedback on problems so they waste little time trying to learn material when they don't have the pre-requisite concepts. Students also frequently pass section mini-exams and enjoy feelings of accomplishment. These positive feelings help with learning.

Compare that to the mastery learning studies performed. The studies gave exams once a week or once every 4 weeks. A student with a crucial misconception will struggle for weeks before the they finally understand the content. During that time, they feel frustrated and unmotivated.

We need a good study of mastery learning.

We also need researchers to design their studies better.

IDEA: A new kind of journal with an open study design process. Researchers submit their study proposal, experimental procedures, example raw data, code for cleaning and filtering the raw data, code for statistical analyses, code for generating tables and graphs from data, and a paper template that includes different conclusions based on the values produced by the code. The paper template pulls in the tables and graphs generated by the checked-in code. All of this content is public. Anyone may register an account and provide feedback. Vetted researchers volunteer to review the proposal and code. They receive credit in the resulting paper. When reviewers give LGTM, then the journal and researchers commit to publishing the paper, regardless of the results, and before they have done any experiments. A separate LGTM is required from an experienced statistician. The code includes assertions for sample sizes and valid data ranges.

The researchers must record video of themselves as they perform the experiments. They must also record raw data from their instruments. They must upload these recordings and raw data. The reviewers must LGTM the recordings and any PII redactions. The researchers must get LGTM for all changes to the code and paper template. The journal's servers execute the template and generate the final paper. When someone later discovers an error in the analysis or code, they can file a ticket or send a pull-request with a proposed change. The researchers commit to reviewing every issue and PR within a time limit. If they fail to do that, then the reviewers must handle it. If the reviewers also fail to do it, then the journal assigns another qualified volunteer as a new reviewer to handle it. After making a change, the system generates a new version of the paper.

Anyone may "star" the paper and receive notifications whenever it changes or there is a change to any of the papers it references. If a paper is withdrawn, the system automatically adds warnings to all papers that reference it.

Thank you for the link. There is a lot to take in there. (Perhaps the author could have provided a computer-based tutor ;-) )

A relevant quote from the work you cite:

> The history of the educational research literature is one plagued with low quality small sample size studies that were done decades ago, with less work being done now. It can be that now researchers are focusing on studying other instructional methods. Still, the fact that most large RCTs tend to find little effects should make us have a sceptical prior when presented with a new educational method.

Khan Academy

I've seen middle schools integrate Aleks (https://www.aleks.com/) online learning along with regular math classes. I don't have the data to know how successful it was at bringing the lower tail of the distribution up, but it was great for kids who were smart and motivated by the material. Some were even able to complete two years' worth of curriculum and test out of their next math class.

I had it for a class in college and it was an absolute nightmare for me. This was from someone well known in mastery learning so I don't think it was poor implementation.

The only thing I learned was empathy for people who struggle that way with the typical mode of instruction (which works very well for me).

Schools are set up for the convenience of the people who work there. Mastery learning is a nightmare. Differentiated learning is really hard because you have to teach what amount to three classes at the same time. Mastery learning is worse because you have to try and teach as many classes as there are students. I exaggerate but you get the point. I know of absolutely no examples of schools using mastery learning. The school I'm in moved to streaming English classes by ability for grades 6,7,8 (Foundation) into A,B,C,D and 9 into A,B,C. Teaching the Foundation classes works ok but I can see why so many teachers hate streaming. Teaching lots of low ability/motivation students is unbelievably dispiriting. Low ability students you can at least help but if you're teaching them something they don't care about, like most teachers do for most of every class then low motivation students are hell to teach.

Schools are mostly child care and ranking by academic ability. Education is secomdary. Mastery learning is really hard and would reqire a wholesale reorganisation of schools. That's a large part of why it isn't done.

Yet despite this being completely obvious to anybody who has any experience with tutoring, I see people rolling out the same old discredited hypothesis of genetics having the largest influence on academic achievement in every HN thread tangentially related to intelligence.