Bloom's 2 sigma problem | Hacker News Reader

Bloom's 2 sigma problem

en.wikipedia.org

57 pointsgballan18 comments

We have implemented this methodology at Lambda School (YC S17 - https://lambdaschool.com) with staggering results. We only get paid if our students are hired making more than $50k/yr as a software engineer or data scientist, so we’re more motivated than most to try things like this, even if they’re expensive.

First, mastery-based progression. At the end of every week we have a “sprint challenge,” where a student builds a project using everything that was learned that week. If they didn’t grasp the material, instead of pushing them forward, they stay back and repeat that week until they have mastered it.

Of course, this is very expensive to do, and therefore will never be done in most classroom environments (you erode economies of scale), but we find that it’s worth it.

The results from that alone have been astounding. We don’t have a scientifically-designed study that shows a standard deviation increase (too many other variables are simultaneously changing), but the difference in student competence and ability at graduation was jaw-dropping.

The next aspect is probably an order of magnitude more expensive to implement - individual tutoring. We obviously can’t pair every student with a tutor all day (our instructors are experienced engineers and therefore very expensive), but we keep every student in a group of 8 students with 1 TA (a student who is at least 4 months ahead, performed well, and stays on for a short contract). Importantly those students are happy to be paid to review the material, and cost about 1/5 of what an experienced instructor does.

In such a small setting a TA can be there to individually work with a student (though for a little less than an hour a day).

Again, I don’t have any good way to scientifically prove this, but the level of ability our students have now at the end of our 30-week curriculum (which can become longer due to mastery-based progression) is incredible.

If you’re an engineering or hiring manager and want to try it out shoot an email to hiringpartners@lambdaschool.com and ask to interview a student who had never programmed before the 30 weeks to see what they can do.

I thoroughly believe Bloom’s 2 Sigma Problem is the most important study done in education... ever.

How can you have great results but not be able to prove it? I see this perception coming up again and again, and it has, in my experience, always been due to 2 factors: the time has not been taken to determine a study design; or the improvement doesn't exist.

I'm not saying your method doesn't work, but surely if it is your business you could take the time and money to prove it (i'm in a similar situation with mine, although in the health space no-one trusts you unless you can publish, which raises a whole other set of problems - ie shitty studies)

We have had a huge swing in results, we can show that, but we’re simultaneously improving and testing a lot of things across the entire company (we’re 18 months old and are iterating in everything quickly), so it’s not like we have a well-designed study that isolates this as the only cause.

We could do it now I suppose, but you begin to run into ethical questions that don’t mesh with our vision: “let’s give n students a subpar experience to prove a point” would be a tough sell.

have you thought of doing a crossover study and reducing fees for a portion of a cohort? If your method has results, then you should be able to catch them up, and then you can go to market with real numbers (alternatively, it might hint towards class sizes or other changes you can make to maximise efficiency)

I’m sure you could find a way to prove it’s better; I’d rather just do a great job for everyone and let the results speak for themselves

Not to harp on about this too much, but that is exactly the problem: results that speak for themselves but don’t have any rigour are sham results and leave you open to accusations that you are simply using marketing flash or selective statistics.

Now, a real study can open you to this as wel, but it is harder when you have some ostensibly objective results.

There is no such thing as achieving results unless you can prove it, it doesn’t make sense

> there is no such thing as achieving results unless you can prove it

That’s not true. Things happen regardless of whether or not they happen within the guidelines of a scientifically designed/valid study.

Students are getting better. Was that correlation or causation? I’m not going to force students to have a mediocre experience to prove it.

In other words, ‘just trust me’. Your business is not self evident

Bloom’s 2 sigma problem was a fascinating study. Regarding the tutoring aspect, at College Connections we want to take this a step further. We use our in-house-designed software MentorMatch to connect students and tutors based on the strength of their personality compatibility. It has been said that “The mind is not a vessel to be filled, but a fire to be kindled.” This echoes anecdotal evidence that students enjoy their work more when they get on with their teacher. Furthermore, we will reduce the cost of private tutoring by targeting schools recent alumni who are happy to tutor students in their schools for a lower rate than professional private tutors. We are currently hoping to conduct a large scale study with few schools to see whether personality-based tutoring matching can actually yield results beyond two sigma! If you want to know more, check this out: https://medium.com/@maxdunhill/matchtech-white-paper-4a6e4e0...

That rings true. At one point I came to the conclusion that high attrition rates are first and for most an indication for lacking training / teaching quality and not difficulty of the course or trainjng program. As one (former?) Pararescue instructor stated, 30% always drop out, 10% always pass and the rest is up to the instructor (unfortunately I lost the source for that...).

The key thing that self studiers don’t realize is that you don’t know what you don’t know. And frequently those things are very subtle but build up over time. Luckily with the internet, any motivated student can find great teachers to help them.

That's true, and the resulting basic mistakes can be extremely hard to get rid off. Like bad foot work in boxing, once you do it you have ahrd time correcting it later.

That’s a good example. Piano is also a classic foundational problem

I was a teacher. Before doing my teacher prep program, I tried to gain relevant experience tutoring after school. When I actually started teaching, I realized very quickly that it's a completely different ballgame.

Teaching isn't just a one way upload of knowledge. It's an interactive process and feedback from the learner is extremely important. Not just for correcting misconceptions, but also for motivation, partnership, and dialog. One-on-one, the loop is very tight.

That's one reason I'm skeptical about technology making a huge impact any time soon. Our technology is just really, really far away from being able to stimulate tutoring.

Technology as a replacement, no; as an augmentation, definitely. A computer could be a powerful tool in improving tutors effectiveness. I just saw a paper on using DNN for generating questions from text which could be used for interactive flash cards (spaced repetition, quizes, etc).

Even for augmentation, I'm bearish, short term. We'll eventually get it right, but not before any kids I have are through schooling.

I think the more exciting takeaway from this is the Mastery Learning method causes a one sigma gain (from 50th to 84th percentile) for free whereas private tutoring lets us gain another sigma (from 84th to 98th) for 10x-15x the tutoring cost.

There’s also something else that’s driving my skepticism further. We know that deliberate practice works [1], and the possible effect size is much larger than two sigmas (think more like from a good chess player to grandmaster [2]). However, at this point the effects of both deliberate practice and tutoring start to blend in too much. The mastery learning aspect already seems like a methodology similar to deliberate practice. Does tutoring cause the additional sigma by letting more deliberate practice happen, or by some other vices? If we’re stuck with a 30-pupil classroom as a tutor, is it possible to extract those vices from private tutoring to help our class succeed?

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3102783/

[2] http://slatestarcodex.com/2017/07/31/book-review-raise-a-gen...

Mastery learning is not costless. The cost is time: time for the student and time for the teacher.

Reply on news.ycombinator.com