I think your final observation is the important part. These studies cost a lot of (often scarce) funding resources. We probably shouldn’t spend the money on hypothetical strategies that cannot be practically implemented.
From a practical point of view, we necessarily want to know if there is a way to distinguish between teachers at or around an average level, since this is what necessarily applies to most cases. (Research on MOOC approaches, video lectures, etc., could be a separate strand of experiment though, to see if it can leverage the top decile to a broader audience).
This study is saying (modulo the usual limits of generalizing from just one study), that teachers a little above the threshold don’t cause compounding improvements in outcomes over a ~5 year period.
But as always it’s rife with possible issues.
- maybe bonus incentives weren’t high enough to induce any different effort?
- maybe thresholds for distinguishing good and bad teachers were not set right, or need to be dynamically adaptable to different circumstances
- maybe there were just confounders, like the mentioned school merger, making student performance primarily causally driven by non-teacher factors over these years in these locations
- maybe the particular student cohorts just respond differently to teacher-based treatments than students in general would
- maybe students or teachers are affected more by social engagements than learning activities, so that the firing of a beloved, but ineffective, teacher causes a student to remain underperforming, or even resentful of a new, but high-performing teacher.
Whatever the case, this single study does not really offer enough to conclusively (or even suggestively) say that rewarding good teachers or firing bad teachers has no effect.
It had no effect in this case, with this methodology. Would need to see replications and variations of the type of experiment before aligning any beliefs about budgets or teacher incentives towards this result.