Comment by theothermkn | Hacker News Reader

Comment by theothermkn | Hacker News Reader

One of the canonical motivating sketches for total quality control, developed by Edwards Deming et al, is a mythical factory with three production lines, each with their own line manager. The lines are performing equally well, staying within a bell-shaped distribution about approximately the same mean. (Widgets per hour, or defective unit rates, or what have you.) As factory manager, in an effort to improve the situation, you decide that, at the end of each month, you will promote the top performer and fire/demote the laggard. Congratulations! You promote and demote/fire a line manager every month for six months. However, you notice that the performance of your lines has remained the same. What happened? You were essentially using the performance numbers of your three lines as a random number generator. The system was in control, producing no results outside of three standard deviations. You promoted and fired at random.

In order to improve the output of a system, you have to understand it. You can’t just blindly incentivize on the randomly distributed results. If this research leads to a more humane work environment for teachers, freer than before of misguided incentive programs, then that will be a good thing, indeed.

The problem with the analogy is that the quality of the output of the line is not just the workers (i.e. teachers), it's the quality of the materials going in (i.e. students).

Good neighbourhoods with low crime and stable, happy family's produce decent students and decent schools.

Not just a money or 'quality of teacher' thing.

No doubt some teachers are a little better than others, I personally believe that it's not that important.

Keeping families employed, stable, keeping gangs at bay, having good role models, low crime, low violence/stress for kids ... these are the big problems.

Your comment assumes that educational achievement is a "randomly distributed result". It may or may not be, but you can't just assume it.

Everything is a randomly distributed result unless proved to be affected by a specific input. That's a fundamental assumption of statistical process control.

> Everything is a randomly distributed result unless proved to be affected by a specific input.

Nope. The former is invalidated by the latter, if you take the statement at face value. Everything appears randomly distributed...oh wait that isn't true either as you always have ranges and generally 2 axes (one being time). Most importantly, this doesn't contribute the conversation in any way. -1

What holds true is that the teachers' effect on their pupils' educational achievements is not under full control of the former, so rewarding and punishing them on that basis alone is ineffective.

Edit: grammar.

Yup, quality of the teacher might be one minor factor among much more major ones (a child’s domestic situation, etc.)

This seems to be a semantic argument with “full control” implying 100%, whereas in reality it could be 98% or 76% or 59% or 11%, who knows.

What’s the actionable item here? Should we fire all teachers and replace them with hamsters who would coincidentally also not exercise full control?

In the sense of Total Quality Control, a process is "under control" or "under full control" when the output of the process conforms to a statistical measure, such as every data point falling within three standard deviations of the mean. To a first approximation, anything outside of that range implies a "special cause of variation," as opposed to a "common cause" of variation. In this case, think of a special cause as something like a layoff at the major industry in town, causing something like stress at home or missed meals. Those would affect the performance of students, outside of the "common causes" of the usual variations in ability, interest, temperament, and so on.

So, no, the answer is not to throw away the experience and insight embodied in the faculty. The answer is to try to understand and remove (or compensate for) the special causes of variation in the population of students, as well as the special causes of variation within the faculty. One of the other things Deming said that I find very insightful is that the interesting thing about geography is not where the borders are, but why the borders are where they are. Applied here, the interesting thing about the variation in grades or test scores is not the variation itself, but why the variation is there, and why is it shaped the way it is. These 'why' questions only have answers that are close to the data and contingent upon the systems, in contrast to blanket incentives that try to sweep an understanding of the system under the rug of punishment and reward.

I don’t think the parent assumes it is random, but merely argues that you need to show that educational achievement is highly correlated with the students’ teacher before using educational achievement as a proxy for teacher performance.

Is educational achievement closely correlated with some teachers and not others? Is the correlation confounded by other variables like the student’s family income or choice of breakfast cereal?

The environmental factors are huge - my wife used to get students who rarely slept two consecutive nights in the same place - but another challenge seems to be threshold effects. Below a certain level the teacher has little control (e.g. a hungry kid needs lunch more than the best teacher) and over a certain level the kid has so many positive influences (high functioning affluent parents, tutors, competitive peers, etc.) that they’re going to do well with all but the worst teachers. The area where a teacher has the most control seems to be in between, with the kids who could coast but will do more if pressed and may not have other encouragements.

Most of the political discourse tends to treat all of those situations as the same.

Insightful!

(Growing up, my classmates were mostly in the "sufficient positive influences" camp, and yeah, the teachers who had big impacts tended to be the really bad ones who destroyed momentum/interest in a topic for one or more of us. There were also high-impact inspirational teachers - but they were more of an "alter the trajectory of what someone wants to do in their life" sort of thing, which isn't really measured by standardized tests.)

100x this.

There are bad teachers. I've encountered a few that few students could learn from. Isolating this effect from the effect of the outside factors would be very hard, though.

> It may or may not be, but you can't just assume it.

It's not random, but it's not primarily attributable to teachers.

Reply on news.ycombinator.com