Hacker News Reader

Comment by kenjackson | Hacker News Reader

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend.

In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- but even a slightly longer look seems to indicate there is a positive correlation in the data. His averaging that he does at the end, IMO, makes it clear.

View on HN

You're missing the point. The author agreed that there is a slight positive correlation in the data. But the averaging makes it look like there is a strong positive correlation in the data, which there isn't.

In other words: without the averaging, you are seeing the full implications of the data: slight positive correlation, but a lot of scatter--i.e., not much meat there. With the averaging, you are seeing only the positive correlation, with all other information filtered out--i.e., you are seeing only part of the picture, and it's the part that, surprise, surprise, makes the person who ordered the data collected look good.

Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all).

Sure, you could add a volatility measure as well, but the trend is clear -- you'd rather have your child by a teacher with a high teacher score than a low one. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile? Of course there's some volatility, but over time, it's seems pretty obvious to me.

The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy

Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful.

Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile?

If you base those percentiles on a metric that has a very low correlation with performance, then there won't be much of a difference in outcome.

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality?

Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality?

What alternatives do we have? Do you know of a more accurate metric?

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here.

The status quo in teaching is to evaluate teachers based on qualitative observations from management, same as it is in most other industries. As the same blogger points out in another post http://garyrubinstein.teachforus.org/2013/01/13/50-million-3... "Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations. As this is what many districts currently do and since this report is supposed to guide those who are designing new systems, wouldn’t it be scientifically necessary to include the existing system as the ‘control’ group? As implementing a change is a costly and difficult process, shouldn’t we know what we could expect to gain over the already existing system?"

The status quo also has the advantage of not being ridiculously expensive and complicated. Ask Bill Gates to release the data if you're curious about its performance, but given what I've seen, I'd be shocked if it was terribly different than these complicated new metrics.

One alternative is to not use these flawed/complicated/expensive metrics.

So replace a metric with 0.1 correlation with one having 0.0 correlation?

If you are advocating that we should use the school principal's opinion rather than VAM, why do you believe opinion is superior? Do you have evidence that principal's opinion has a higher correlation with student outcomes than VAM?

So replace a metric with 0.1 correlation with one having 0.0 correlation?

Two problems with this:

1. The other metric almost certainly doesn't have an 0.0 correlation, but we don't know that for sure since the data wasn't released for whatever reason.

2. The metric with 0.1 correlation (or whatever the number is)... keep in mind the context. What is the correlation with? Test scores, something that can be and is often rigged, and is at best tangentially related to the ultimate goal which is inherently qualitative in nature. I think the comparison to programming is instructive. If you had 1/3 of your salary determined by the LOC you wrote or the number of bugs you closed, you would game the system and maximize your salary, even if you weren't providing value by doing that and even if those naive metrics did very weakly correlate with performance in some big study prior to implementing the monetary incentive. What would be gained in that scenario?

One of the points made in the blog I linked to is that it's unfortunate that they didn't release that data. And as I said above, it's not straightforward to assume that these weak correlations are actually meaningful in practice.

The 0.3 correlation described in the article is the correlation between a teacher's VAM score in a single class last year and this year.

The author is arguing that because the measurement is noisy, we should ignore it. This is silly - it just means a single class-year's point estimate is noisy, and multiple class-years must be combined to form an accurate estimate.

If you had 1/3 of your salary determined by the LOC you wrote or the number of bugs you closed, you would game the system and maximize your salary...

Indeed - if 1/3 of my salary was determined by the number of kids in my class who can read better than when they are predicted to read at this age, I'd definitely try to make sure that reading skills improved.

Conversely, if a stupid metric such as student/principal opinion were used, I'd focus on jokes and friendliness over education.

(Well, actually I didn't back when I taught. But that sure didn't help my student evaluations...)

The author is arguing that because the measurement is noisy, we should ignore it.

Not quite..

Keep in mind that "score on a reading test" and "reading skills" are not the same things. For instance, one would be significantly improved by spending valuable class time teaching students tricks about how to succeed on the standardized reading test. It's similar to the incentives you'd get by determining salary by LOC - even if LOC correlates with performance, basing pay on it encourages all kinds of nonsense that doesn't actually benefit anyone.

So in that context, we have a metric that very loosely correlates with this clearly flawed marker of success. And we're supposed to spend ungodly sums of money implementing this strategy? Come on.

Conversely, if a stupid metric such as student/principal opinion were used, I'd focus on jokes and friendliness over education.

Well, they do propose to use student and principal evaluations..

Keep in mind that "score on a reading test" and "reading skills" are not the same things.

True. Score on a reading test might be lower than actual reading skills due to a lack of test taking skills.

Then again, I can't think of any measurable quantity more highly correlated with reading skills than scoring high on a reading test. Do you have evidence that "principal's opinion of teacher" is better?

...we have a metric that very loosely correlates...

You are confused. GR shows that year on year correlations are weak (0.3), not that correlation between reading test and reading skills are weak.

You are correct that a certain nonzero amount of teaching test skills will be effective at improving scores. I see no reason to believe this amount will be large - do you?

Again, standardized tests are not the only gameable metric. Principal/student opinion is as well. Why do you believe they are less gameable and more accurate than directly measuring student ability?

It's not that I think this is the only flawed metric or that there are other methods that are quantifiably better. It's that we've spent ridiculous sums of time and money on this complicated new metric, despite some pretty clear arguments against it, and we still are very far away from actually proving that it is better than the status quo, even if that status quo obviously falls somewhat short of perfection.

It's like.. how much money and effort do you think is worthwhile to spend on something with such uncertain prospects? There are tons of great grant proposals to funding agencies which are rejected. Maybe we would be better off funding some of those. Opportunity costs exist, and it seems like undue sums of money are being wasted on this stuff.

Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations.

It's not that conspicuously absent as they don't use 100% weighting for anything. That said, they do attempt to maximize ability to predict state test scores, and in that optimization classroom observation plays the smallest role of the three metrics (2-9%).

Given they appear to have to done the analysis across all weightings, I seriously doubt they'd hide this data.

I do find it odd that teachers would rather be judged based on one or two people observing their classroom and ignoring actual student output, rather than raw numbers.

As a developer I'd much rather be evaluated on some metric I could optimize for (I think some metric measuring feature value/bugs/fix rate/etc...), rather than just my manager watching me code/debug a couple of times per month.

I do find it odd that teachers would rather be judged based on one or two people observing their classroom and ignoring actual student output, rather than raw numbers.

Okay, I really don't understand this mindset. You'd really prefer it is, say, 1/3 of your salary was determined by the number of lines of code you wrote? Or the number of bugs you closed? Or some equally inane metric that only loosely correlates with your actual performance and can be easily gamed?

What am I missing?

You'd really prefer it is, say, 1/3 of your salary was determined by the number of lines of code you wrote?

I was a quant trader for a while. A considerable chunk of my salary was determined by profit, hardly unreasonable.

Measuring developers in other areas is difficult because of heterogeneous goals - last year I built a search engine, this year I'm statistically tracking user behavior. Hard to compare one to the other.

Education does not suffer this problem - last year a teacher taught 30 kids to read. This year she taught 28 kids to read. The goal is always maximizing the fraction of kids who can read.

I don't think it's obvious that these teaching metrics correlate better with teaching ability than LOC would correlate with programming ability. I'd expect weak correlations in both cases. I'm just speculating at this point, though.

The point is, you shouldn't worry too much about teacher performance.

If it's only loosely correlated, measuring value-added then punishing / rewarding teachers based it won't be very accurate. It's like lines of code - programmer talent is probably weakly correlated with lots of lines of code, but you don't want to reward or punish them based on LoC because it will lead to pathological behavior.

You should look at other things to alter (which are less easy to game once you start trying to control them), like class size, course materials, assessment style, how the teacher actually teaches, etc.

You are missing the connection between these statistics and school HR policy. Sure there is a small positive correlation, but at what cost should we extract this benefit? Given this data, should we use value-add as a major component of teacher evaluations, and ultimately compensation/terminations?

Giving employee evaluations that are 5% performance and 95% random chance is a recipe for an organization that no capable intelligent person would consider for a career.

What form of evaluations do you believe perform better than VAM, and why?

you'd rather have your child by a teacher with a high teacher score than a low one

Not necessarily. The very weak correlation means that the difference in this score only accounts for a very small amount of the difference in outcome; and even that claim assumes that the observed correlation is evidence of a causal link, which is not necessarily the case. You can't usefully conclude anything from this data except that more investigation is needed.

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting.

And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money. I find it really difficult to give any criticism directed at Bill Gates' foundation any credence. And that not knowing much at all about what he does.

Maybe people would like Bill more if he did a Larry Ellison and bought a yacht instead.

Bill Gates's position as a famous wealthy guy means he has more of a responsibility than the rest of us to not put out misleading graphs. Even if he is spreading misinformation with his own money.

But Bill Gates is not putting out the graphs.

This looks to me like someone trying to make the report look good so that Bill Gates will get positive press, think this was worthwhile, and fund them again.

Many would argue that Bill came by that money via illegal and unethical behavior, paid for by millions of consumers, so it is more like tax money than his own money.

Luxury taxes crimped the yacht industry and many workers. Fiscal conservatives would appreciate some support for the yacht industry.

People are arguing that he is using his own money to actively damage society. I'm not sure if I agree with that, but to say you can't criticize someone for what they do with their money is ridiculous - as an extreme, he could use his money to poison people, or to lobby for life sentences for marijuana use.

I said Bill Gates, and not someone. And while I don't know how people think he's damaging society, if the article being discussed here is representative then I remain unconvinced. This is not an invitation to enlighten me. I'm not wasting my time reading yet another opinion, assumption or half-baked arm-chair allegation. The investment dollars and opinion of the likes of Warren Buffett are a much more compelling endorsement to me than (I mean this respectfully) members of HN.

There's a lot of distrust for any billionare meddling in the education system, and with good reason - they tend to push faddish ideas with questionable evidence, like this one, and somehow manage to convince the media they've "solved" the problems with the education system. Then the state wastes a whole bunch of money on the latest fad.

Also, that's probably why the headline calls it a "lie". This is something of a pattern.

"Whereas to me it looks like it removes visual noise to show an actual trend."

Except for that in this case the noise is more important than the trend. Think about it, if you're firing or sanctioning perhaps 30%+ of teachers each year for no reason, then only complete morons would go into teaching.

It's the same as airport security, where a .1% false positive rate is unacceptable, whereas a 75% false negative rate is just fine.

Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%. Which is a bit different than 30%.

So I'm thinking they're not too worried about this.

"Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%."

This is comparing apples and oranges.

Did you transpose false negative and false positive in your statement about airport security? I think airport security would much rather have false positive (i.e. This guy has something suspicious, let's do further checks, oh, turns out it was nothing) vs. False negatives (i.e. This guy is clean, oh, turns out he wasn't and blew up a plane).

No, that's correct. The issue is that the overwhelming majority of people aren't terrorists. So even though it's worse to let a terrorist on a plane than it is to ban someone who isn't a terrorist from flying, in aggregate the harms of banning non-terrorists from flying become greater than the harms letting a few terrorists fly even when the false positive rate is very small. Bruce Schneier has a good explanation of this somewhere on his blog. (Actually, this is one of his pet issues, so there are probably dozens of blog posts about this.)

I agree 100%. A scatter plot, with a low but very obvious correlation, shown for two years only seems to me to be absolutely fantastic evidence that progress is being made.

Yes, that is what the people at Lehman Brothers were saying about the Mortgage market for some years.

But anyway, each one is free to reach his own conclussions.

+1.

The blog author saying there is no clear trend in his random blob of points is misleading. The trend is there, it is just difficult to tell the density of points in his scatter plot because of the marker size and plot size.

Reply on news.ycombinator.com