Hacker News Reader

Comment by Dwolb | Hacker News Reader

I did some market research back in the day and there was a lot negative opinions on even attempting to measure performance based on patient outcomes because “each case is unique”.

To me there were two things at play: possibly the sample size of number of patients doctors see with the same problem might be too small and there was genuinely a lot of fear that the doctor’s skill could be boiled down to protocols and statistics.

I think the biggest hurdle toward improving patient outcomes will be alleviating the fear and then after that figuring out how to give better context to the data.

View on HN

As someone who works in education, I find it interesting that "each case is unique" hasn't prevented outcome based metrics in education.

It really should but I think thats a much more difficult problem to politically explain why every case is unique!

A very simple hypothesis as to why doctors can argue this effectively but teachers cannot is that teachers are not as powerful a lobbying group as doctors.

Doctors have effectively been able to defend their "turf," from hostile encroachment, while teachers have not, not because the situations do not contain substantial parallels, but rather because doctors are politically strong while teachers are politically weak.

Opposition to most doctors is low, they aren't being forced to be rated by their clients, whereas most teachers have a weak union in a district with poor funding, and thus they have to contend with overloaded classes that result in poor educational outcomes that lead to most students not going on to do much with their lives.

Its much easier to punch down than it is to punch up!

The term "Gods in white" exists for a reason, many people ascribe a lot of authority and knowledge to physicians and doctors in general. Doubting your doctor's diagnosis is usually considered a rather odd thing to do and if it's a doctor of psychology it could even be interpreted as a symptom.

While with teachers it's kinda the opposite; Even tho their whole job is to know and teach things, many people have a way easier time disagreeing with them straight out of principle.

Wonder how much of that boils down to socialization aka in what contexts children are introduced to these professions?

I find it interesting that teachers have a union and doctors have a professional association. I think that distinction goes a long ways in the comparison.

In education, you're effectively looking for long-term financial success of your students, and you search for proxy metrics that predict it. Most of the things you need to control for are the same socioeconomic factors behavioural scientists are working on in other fields. Also: Since almost everyone goes to school, there's a lot of easy-to-anonymise data.

In medicine, I don't think there's a clear long-term metric (lifespan? financial success? happiness?), and there's at least two nasty selection bias that people generally only see a doctor when (self-report) something is wrong, and that doctors specialise. But let's say you solve those problems: What kinds of questions[1] are you going to ask that are supposed to predict satisfaction with the doctor?

[1]: https://londoncalling.co/2013/02/simple-customer-feedback-id...

I don't see "long-term financial success" as being obviously the right metric for judging education at all. It's worth considering, absolutely! But people choose to forego maximum wealth for other priorities all the time, and often for very desirable reasons (both individually and socially). Why shouldn't the metric have something to do with long-term personal fulfillment and happiness? Why shouldn't it have something to do with long-term contributions to society?

I agree with you that the more abundant data (and reduced selection bias) available for schools makes measurements easier for them than for medicine, but I don't think that it's any easier to define an appropriate metric for judging them.

HN really likes to talk money over everything, in spite of how money is merely a means to an end.

I think a better metric would take into account overall health history, years paid into Social Security (as a proxy for income), and food/housing stability. Look at those three factors, and you can get a grip on how a person's life has gone in a way you can compare to others.

I sure hope my kids` teachers don't share your views of their success metric.

Isn't this a common objection by teacher unions when pushing against using metrics for retention and compensation?

They may be right, but they have a very strong prior interest in this matter (that there can exist no objective measure of a teachers competence), so not the best arbiter of objective insights here.

I know most of HN hates unions, but teachers unions are usually there to create a better workplace, which ultimately makes for better teaching outcomes.

The difference between a 24 student class and a 35 student class is huge in terms of educational outcomes, yet here in the US we tend to cut spending on and underfund education, either by ripping away stable revenue sources like property tax and replacing that with lottery income (which varies wildly from year to year), or by just slowly reducing the amount of money allocated toward schools year after year.

Hence how you end up with places like Seattle having well funded schools due to local ballot measures, yet a similar school in Eastern Washington will get 20% to 30% less funding, and have abysmal results due to it.

Proper education isn't a technology problem, but an investment problem, and rather than investing in schools today, we would rather lock many of those students up in the future here in the US.

It isn't about "hating" unions, or otherwise, but about recognizing the simple fact that unions aren't magical unicorns, they are interest organisations and the teachers' unions represent the interests of the teachers. Now, those interests are often aligned with the interests of the students, but students' interests do not flow from teachers' interests.

To get back on topic, teachers are opposed to objective measures because if you can objectively tell if a teacher is bad, the union will lose influence over pay and employment security, which is where their power is anchored. Thus, they will of course promote studies that show the failures of such objective measures and try to discredit those that show the opposite. Just like any other interest organisation.

That doesn't mean that I have a problem with the existence of interest organisations, but let's not pretend they are something they're not.

> To get back on topic, teachers are opposed to objective measures because if you can objectively tell if a teacher is bad,

Or that since there exists no such measurement pushing for one is dishonest and betrays other agendas.

Of course. But the whole point is that we shouldn't treat a teachers' union as an authority on whether it does (or can) exist. And no, it's by no means an established fact that it can't.

>but teachers unions are usually there to create a better workplace, which ultimately makes for better teaching outcomes.

Unions exist to lobby for their members; teaching unions represent teachers, not students. All sorts of things might be in the interests of students but contrary to the interests of teachers.

The obvious example is the New York City Department of Education's "reassignment centers". About 600 teachers in NYC are paid their full salary to sit in an office doing nothing. They're not sufficiently trusted to teach due to allegations of misconduct, but neither can they be fired due to strict tenure rules. This system is believed to cost $65m a year. Nobody is happy with the system, but the only way it can meaningfully change is if it becomes easier for the NYC Department of Education to fire teachers.

It's entirely plausible that pupils might be served better by teachers who are brutally over-worked and subject to constant management scrutiny. We know from the private sector that wringing your employees dry is often a very effective way of improving the bottom line. That wouldn't necessarily be the right thing to do, but it simply doesn't follow that what's good for teachers is always what's good for students.

https://en.wikipedia.org/wiki/Reassignment_centers

> Nobody is happy with the system, but the only way it can meaningfully change is if it becomes easier for the NYC Department of Education to fire teachers.

It could meaningfully change if the NYC Department of Education paid their share of arbitration fees, or if the DoE and the union agreed to hire more arbitrators and work through the backlog of cases, or if the DoE got better at reassigning those teachers who have been cleared (many of those in the reassignment centres are teachers who were cleared in their arbitration hearings but the DoE haven't assigned to teaching positions). Partly this is because the DoE doesn't really believe in innocent until proven guilty, and part of that is because they don't trust their own bureaucrats to correctly document genuine misconduct and prove it to the standards the arbitrators require. Fixing that is absolutely on the DoE.

What skilled, dilligent person would want to work in a field that consists mostly of unsupervised interaction with children if one unsupported allegation of misconduct could be a career-ender? That's where you'd be without the arbitration agreement. In much of the rest of the world that's how firing people from any job works - you can't do it without evidence. Somehow those countries manage to be productive. A competent department of education would be able to do the same. Giving an incompetent department more ability to fire people would not be progress.

>Unions exist to lobby for their members; teaching unions represent teachers, not students. //

In the UK I have some knowledge of the work of one of the largest teaching unions the NASUWT.

The members are teachers. They're primarily teachers because they have a passion for teaching. As such their union actively works to improve _learning_ in schools and so lobbies for activity that benefits pupils.

An example. Pupils get a certain amount of money allocated, that money is allocated politically and in a devicive way but does not impact on wages at all, nonetheless the union lobbies for fairer and more transparent distribution of such pupil stipends.

The union does help it's members, but it also does research, shares best practice, and I dare say other things, that help improve _teaching_ too.

They also campaign on social issue - racism, sexism, forced marriage, ... even when it's not directly related to teaching.

Perhaps your characterisation is true of specific teaching unions in USA?

The issue with the Reassignment centres appears to be they can't sack teachers for doing unlawful things without a trial (seems reasonable, presumption of innocence), they lack evidence to go to trial, but they feel the teacher can't be trusted.

What's your solution? Sack people without due process? So sack teachers when there's a suggestion they might have done something wrong? In situations where pupils can easily make false allegations (without repercussions on themselves) that enables pupils to get teachers fired very easily; is that fair in your opinion?

> Unions exist to lobby for their members; teaching unions represent teachers, not students.

And what do teachers want? Educate students to the best of their ability. Yes, I'm giving the benefit of the doubt here that many teachers actually want to do the educate and ain't just in it to "get rich" because if one's sole motivation is making money then god knows there are a myriad of better choices than becoming a teacher.

Imho this representation of unions as "oh so greedy money grubbers who only do it for the money" really irks me, especially considering that the original reason for unions actually being a thing was to counter exactly such "greed money grubbing" ways by exploiting labor in mass when it was still living in serfdom or straight up slavery.

Unions do not just serve the purpose of "making union members richer", even if that has become the de-facto reality in some places. Without unions it would be doubtful anything like workplace safety regulations would ever have become a thing, many standards for healthy and productive working, we take today for granted, are the direct result of unionization in some way or another.

Your numbers indicate the average salary for these 600 teachers is over $108k, I think your numbers are wrong.

I am not OP, but it only indicates that salary + benefits + HR / admin costs + space per person = $108K, which probably equates to a salary closer to $60K.

Yes people have come up with these metrics (mostly bureaucrats with a fetish for anything that looks like data).

As far as I know no-one has come up with a metric that's actually useful for students and teachers.

Teacher-based metrics in schools turn out to have extreme variance.

In cases where the same teacher was subject to multiple assessments (there's a Math Nerd writing for Bloomberg who presented this, I think at TED), there was no correlation between the multiple measurements.

Straight-up noise.

Cathy O'Neil, "MathBabe", "Algorithms are Opinions in Code".

Starting at 2m22s here: https://www.youtube.com/watch?v=_2u_eHHzRto

Their fear (for their own professional standing and income stream) is justified, but the objections to measuring performance are special pleading.

1) "Each patient is unique" just like "each student is unique"...but we still give standardized tests (to the grumbling of below-average teachers and administrators)

2) If the sample size is too small because the disease is rare, there may be a genuine measurement problem. If the doctor or hospital's sample size is too small but the illness is common, then they should be referring those patients to providers/facilities with the requisite experience levels.

If a common disease has not sufficient experience in a hospital I'd argue they should pull in experts from elsewhere and try to gain the experience necessary, otherwise you just end up exporting the problem.

There is substantial argument that our focus on standardized testing is literally ruining the advantages traditionally associated with a Western Education.

But then, perhaps I would be a "below-average" teacher, and my opinion might therefore similarly be discarded as worthless. We'll never know for certain, because I don't make a habit of boarding sinking ships - especially when I'm slated to receive blame for their sinking after boarding them. (And I did at one point very much want to enter the field of secondary school education.)

(Slight modification: I won't board a sinking ship and take blame for it's sinking after the fact without some substantial advantage, such as excellent remuneration, being offered as well - teaching offers no such advantage, except perhaps self-actualization, and I can neither eat nor sleep in that.)

Sure, bad comparison. It's consistent to support standardized rating of doctors but not teachers. Teaching exists to enrich, whereas medicine properly understood only exists to solve problems.

(I find it curious that another poster takes the opposite tack, claiming that teaching exists only to maximize the financial success of students, whereas medicine has a more nuanced end. I confess to finding this position truly bizarre.)

Having worked a little bit on #2, even for relatively common conditions, stochasticity still plays a major role in outcomes.

For example, I have a simple model that ends up showing that an ICU doing everything it's supposed to do (patient isolation, high hand hygiene compliance, perfect diagnosis, etc.) can have a four-fold difference in MRSA infections over a year by chance alone.

For what it's worth, I've heard a whole lot of above-average teachers complain about standardized testing, too. The things that one does in a classroom to optimize standardized testing performance are often not the same things one would do to optimize creative problem solving, or independent learning, or good citizenship, none of which are captured by (existing?) standardized tests. For that matter, the things that a third-grade teacher would do to optimize third-grade standardized test scores aren't necessarily the same things that the same teacher would do to optimize twelfth-grade standardized test scores.

Case mix and case severity are real concerns. Physicians (and hospitals) vary tremendously in each, and often the best physicians get the hardest cases, which, if not accounted for in metrics, make them look bad.

Data are also often tremendously thin -- I've seen profiling efforts over tens of millions of patient-lives in which a given provider may only have a handful of records. That's not enough to draw strong inferences, and year-to-year variance is going to be tremendous.

It also turns out that there are very strong clusterings of patterns around facilities: a senior physician or surgeon can (and does) drive practice, quality, and methods for much of the rest of the medical staff.

The single best predictor of quality I recall was volume of procedures. Practice (and standardisation) help tremendously.

Reply on news.ycombinator.com