Hacker News Reader

> The issue is that they are very hostile to the educational administration establishment that has decided test scores are the ideal measure by which to base this.

The problem with that arguement is that teachers are hostile to every form of evaluation that has been proposed and are unwilling to propose anything that they'd find acceptable.

> Peer review is the ideal.

No, it's not. That's like asking ford to review chevy. It leads to backscratching.

> it certainly will create cliques of teachers, but only rarely cliques of bad teachers.

You clearly don't have any experience with teachers.

> those facts that are deemed convenient and useful by powerful cliques in the state legislatures.

Those facts have the interesting property that they're at least nominally under the control of the folks paying the bills.

anamax, you clearly have some experience here. To add to your points: (1) I've never met an adminstrator that said that test scores were the ideal measure of a teacher, it's just that its the best they have. Furthermore, (2) the tests are made by teachers. If they don't like what is being tested, then change it.

When news came down the pipe that Virginia was going to have a "Standards of Learning" type exam for colleges, the first thing I did was jump on board to help scope (and my plan was to help author) the test (I was physics faculty). I'm no fool: if we're going to give the administration a vehicle this powerful, I'm driving.

On the other hand, when the Standards of Learning for K-12 schools were first implemented, a remarkable thing occurred in education: teachers started talking about teaching again. Until then, every conversation with them was about how bad the administration was, how difficult parents were, how unprepared and unmotivated the kids were, etc. When those tests went in the conversation turned to: how in the world am I going to teach all of this?

How in the world, indeed.

You are also correct that the way peer review is implemented leads either to back scratching or very poor morale. The way to do it is to get folks from one institution to review folks (as a team) from another. That would place a huge demand on the personnel, though, and I think you would have trouble getting it in place.

You get this in milder form in higher-level education as well. If you look at some of the recent debates over tenure, lots of professors will agree that there are dead-weight profs who basically quit working after tenure, and we ought to be able to fire them. But the crucial question is: who decides if they're a dead-weight prof, and who gets to fire them? And will this add a bunch of new non-research work to satisfy whatever process gets invented? Academics have basically zero trust in university administrators being the ones to do that--- the worry is that if tenure's abolished, people will be hired/fired for metrics-based reasons, like citation count and how many grant $$ they brought in in the past 5 years, or even for trendy university-marketing reasons ("I hear X is hot, we need more profs there, not in Y" or "we need more of the kinds of profs who get interviewed on CNN a lot").

It is right that we should be suspicious of people who want to be paid without being evaluated. What other industries operate like this? Even "creative" industries, where everything is subjective, have cold hard metrics in place.

I'd say most good industries don't have cold hard metrics in place, and the companies with more metrics within industries are generally the worse ones. There are engineering firms that have heavily metrics-based evaluations of engineers (e.g. bug-fix counts being directly tied to bonuses), but I don't think I've seen them being called out as good examples. The better companies usually have engineering managers who are actually in touch with what their team's doing, and can give accurate subjective judgments of who on their team is contributing what and what the relative value of their team members is. It's the companies/managers who don't have any idea what's going on in their teams who fall back on metrics to try to answer those questions.

At the simplest level: if the company doesn't meet its revenue and profit targets, I don't get a bonus.

Teachers aren't even evaluated at the level of "more exams passed than last year" or overall results relative to comparable schools.

We've seen problems with compensation linked to short-term company performance. Test scores and most metrics like you're proposing have similar problems. Nobody cares about test scores; we care about long-term success of students into adulthood, and it seems that we have no idea how to predict that.

As the article illuminates, we're not even sure which aspect of teaching is important for it.

This is the real issue. I have absolutely no problem with rewarding good teachers and sanctioning bad ones. The crucial question is how good and bad will be defined. Standardized testing is absolutely not the way to do it.

If you mean that a teacher's salary shouldn't be a simple function of test scores, then sure. If you mean standardized testing shouldn't be used to evaluate children, or that evaluations of students - even relative ones - shouldn't be used in evaluations of teachers, then can you please explain what you have in mind as the alternative?

I'd go with subjective judgment by human observers, which is fallible but I think reasonable enough, and harder to game, when you're looking for large differences. We're not really talking about distinguishing 50th-percentile from 60th-percentile, but trying to pick out "absolutely horrible at teaching" (get rid of them) from "fine at teaching" (keep them) and "amazingly great teacher" (give them raises/bonuses). I think principals might actually have a good estimate of that even today: the worst teachers, at least, are usually something of an open secret. If principals ideally stuck around longer (instead of the current musical-chairs that seems to go on), they'd have even better estimates.

Sort of the same way that companies do it, really. Engineering firms, or good ones anyway, rely more on subjective judgment of management ("so-and-so is a [terrible|great] engineer") than they do on code metrics. I think they'd have some of the same morale problems if they were heavily metrics-based as well: imagine if every year your company ran a script over their VCS+bug-tracker with whatever metrics were currently in place, and fired everyone who scored in the bottom 5%.

Lines of code produced is a terrible metric of software progress and especially software quality. By comparison, a well constructed test is an excellent metric of learning.

Also, have you ever written software for the government? I think if you had, you'd yearn for an objective evaluation procedure as accurate as standardized tests. For your proposal to work, you'd have to carefully set up incentives for the judges and try to figure out how to keep the politics to a minimum.

If "teaching to the test" is a bad thing, that just means you have a bad test. With varied enough questions, the easy way to prepare for a test is to understand the material. I agree that teachers shouldn't be judged directly by the scores of their students, because not all students have equal background and ability. But any time you have an opportunity to measure something objectively, jump at the chance. That goes for software, too.

Well, "well constructed" is the tricky part. It's not even clear to me what one would look like. K-12 schooling has a lot of goals, most of which don't involve literal transmission of information, but boil down to something like, "we want as many kids as possible to be normal members of society, with decent jobs, some sort of civic engagement, not engaging in criminal or shady behavior, and able to handle their affairs". Ideally we'd also like some subset to be brilliant, although it's not clear how much K-12 education influences that.

You can judge most of the goals in retrospect, say 20 years out: what proportion of kids ended up in jail, what proportion ended up with good jobs, what proportion are informed citizens capable of reading the news and participating intelligently in the political process, what proportion are able to handle their personal finances, nutrition, health-care, etc. successfully, what proportion won prestigious awards or started successful businesses, and so on. But writing a test for 18-year-olds to take that predicts all that accurately is pretty hard, and as far as I know, no tests have shown even plausible evidence of their predictive power in that respect, especially when you control for demographics.

Again, I'm not saying that 100% of teacher evaluation should be based on tests. If you want to evaluate teachers based on the moral character they are building in students, fine - that's not what I'm advocating standardized tests for. They're for measuring "book learning" and they're pretty good at it.

>If "teaching to the test" is a bad thing, that just means you have a bad test.

Look at it this way: We do not want students to all know the same things. We want to increase the amount of learning, but ideally, only 30-50% of the knowledge kids learn should be held in common. In a perfect world, the remaining 70% would be different from classroom to classroom. Varied experience is crucial to making our society function better.

So if we take varied experience as a value we want to encourage in our classrooms, you cannot have a good standard test, because the standard is more amount and quality of knowledge than a predefined notion of what knowledge is necessary to get by in the world.

And likewise, because this is how I value it, I would prefer that standardized testing be around one-third of the evaluation score, if not 20%. I want teachers and school administrations to be free to take big risks like teaching everyone computer science instead of math - because teaching a well-thought-out course of study is not a risk. Classrooms should definitely do instruction in ideas of the past (history is always vital), but more than anything they should be incubators for new ideas.

Well, by your own admission, 30-50% of knowledge should be held in common. In early grades, the number is probably more like 90%. By high school, sure, there's plenty of room for diversified knowledge of literature and history, but there's also a core of math, logic, and reading that everyone should know. Why not test this stuff? Standardized tests are objective.

Now, the current system focuses on meeting minimums, and most standardized tests quickly hit a ceiling. I'd much prefer to see tests that attempt to gage the entire breadth and depth of student learning. Practically, that's going to require different questions asked of different students, or you'll end up with a 10,000 question exam. But I still consider it standardized testing. The SAT and AP exams are great - we should do more of that earlier.

>Why not test this stuff? Standardized tests are objective.

Sure they are, but to what degree do they really reflect on the teacher? A teacher can be doing a fantastic, patient job with a student, and make zero progress all year with a student. That work is meaningful, but its effect could take years to show up. Can we measure this? Sure. But it will be very error-prone, and ultimately less effective than just asking the other people in the building who is worthless (everyone knows except the worthless themselves.)

We're not looking to find the best and brightest - we as a society have made the decision that we're not going to pay that much. We just want to get rid of the unequivocally awful.

Simple solution: create a set of standardized tests, on Math, History, etc, and require students only to pass a subset of them.

If you object to standardized tests, what do you propose we use? Non-standard tests? The opinions of assorted cronies?

Well, that's sort of like saying we should use lines of code as a metric, because what else to we have to go on. The opinions of assorted developers?

There needs to be some attempt at normalization between teachers and school districts, of course. But there's room for qualitative measures instead of quantitative ones. Especially when the tests seem (unintentionally) designed to punish teachers for teaching poor/minority/ESL students.

We have much better metrics for developers than lines of code - bug fixes/feature requests weighted by importance, P&L, etc. Do you have a better metric for teachers than standardized tests?

Especially when the tests seem (unintentionally) designed to punish teachers for teaching poor/minority/ESL students.

This is utterly false if we reward teachers based on deltas rather than absolute values, that is we compare actual performance to statistically predicted performance and reward/penalize teachers when actual performance exceeds predicted performance.

> bug fixes/feature requests weighted by importance

Isn't there a DailyWTF about a company that did just that and found that the testing people conspired with the developers to invent bugs, locate them, and then collect bonuses for fixing them.

Preventing conspiracies like this is exactly why teachers should be evaluated by standardized tests designed by an independent agency.

You already said you think teachers are overpaid and underworked and your goal is to make things harder for them.

Why should they want to work under a system like that?

What independent agency would you want devising metrics to figure out how good you are at your job while knowing nothing about the specific challenges you deal with?

I believe teachers currently are comped at above-market rates. I would bring them down to market rates. They would work under that system for the same reason most other professionals work at market rates - it's the best they can get.

The independent agency which currently devises metrics to figure out how good I do my job is the stock market. My bonus = % of profits.

When I was a professor, I favored (and suggested to the department) that some higher authority should devise midterm and final exams. It would a) allow us to measure teacher performance, as opposed to student opinion [1] and b) allow comparison of student performance across classes.

[1] To get good student evaluations, all you need is easy tests + jokes + good looks.

You've got market and non-market all mixed up.

Public education isn't a market good, it's a public good. Toilet paper and oatmeal are market goods.

The labor market for teaching is in fact a free market. Any given municipality can pay whatever they want to pay, and get the talent they're paying for accordingly. The fact that teachers are getting laid off in something like 80% of municipalities the last few years means you don't even have to scrape the bottom of the barrel.

Agree on the statements about professors. All the same. If you were working on something more qualitative as part of a team, and not a lone trader, we both know there's no quantitative way to assess your work. Lines of code is a joke and other metrics invariably bring in subjective measures.

EDIT to add: If you're pro-free market, shouldn't it be up to municipalities to choose which teachers they want to keep or fire as they see fit, without some bureaucrat telling them that according to some BS test scores they have to do such and such?

Education is not a public good since it's both rivalrous and excludible. That makes it a private good. The fact that a private good is provisioned by the government does not make it a public good.

Also, the labor market for most govt employees is not remotely a free market. Unions keep out the competition, essentially getting no-bid contracts to supply labor. They play politics to elect politicians who will give them what they want - in many cities the teachers unions essentially control the political system. You might as well call Halliburton's no-bid contracts in Iraq a "free market". It's a "free market", closed to new entrants, with no negotiation between the buy and sell side, but with the buy and sell side colluding to rip off the public.

In any case, just because it's tough to quantitatively measure some development does not mean it's tough to measure teacher performance. Students either know the material at the end of the term or they don't, simple as that.

All the whining about how it's impossible is really just an attempt by teachers unions to avoid accountability. I'm sure Halliburton would love to avoid accountability for their performance in Iraq and elsewhere too, doesn't mean we should take them seriously.

So, by your logic, roads and sewage are private goods? Basically everything except the military? Consistent with a libertarian.

In almost no city does the teacher's union essentially control the political system. The police union is always, always more powerful, for one. They're just more politically engaged, people who are driven by power. Several private unions will be more powerful. And that's leaving out every power broker that isn't a union (representing most of the power).

"All that whining about how impossible it is to measure programmer productivity is just an attempt by developers to avoid accountability." You might consider that you don't know a single thing about teaching in a non-college or lower-class environment. No offense. I don't claim to, either. But I'll at least listen to what every teacher I've ever met says about the matter.

Programmers may routinely be judged by peers, but the entire team is then judged by results. A team that can't ship gets the axe or goes under. The closest counter-examples to this I can think of, doing government contracts, have also been some of the worst software shops that I've encountered.

So what's the analogous mechanism going to be? Are you ok closing entire schools if they have lower than average test scores in their class (whatever this means)? Otherwise, how do prevent the whole system from devolving into politics and cronyism?

This presupposes that a standardized test can test anything of value. I've spent the majority of my last 16 years in school, since the 3rd grade or so I've had no problems evaluating the competence of my teachers, and I didn't need a test to do so.

But how do you know your evaluation is accurate? A colleague of mine was fond of saying that asking your average student what she thought of her teacher was like asking your average seven year old what she thought of her flu shot.

Even if you are exceptional in this regard, the average person will not do well. And I have read a lot of reviews in my life from college students.

Until you know the subjects well, you are only evaluating whether or not you liked it.

This is utterly false if we reward teachers based on deltas rather than absolute values...

That's a big if. From what I've seen, this is practically never done - absolute values are the norm.

I don't have an alternative that I think is workable. And that's why I'm far from convinced that teacher salaries should be tied to performance; I think the side effects of "optimization by proxy" will be severe.

As an aside, pointing out that something is bad does not put the onus on me to invent something better. The sensible conclusion is to not do anything until something good has been worked out. The alternative is to do something just for the sake of appearing to do something.

If students weren't locked into schools enrollment levels and parent feedback would provide a lot of information.

Parent feedback isn't worth nearly as much as you might think. Parents aren't in the classroom, they tend to get one side of things (their child's). Some parent's are objective and some aren't. This leads to plenty of horror stories about parents that assume all teachers are morons and their children are infallible, I've seen it first hand and it's really pathetic. The flip side of that would be the parents of my wife's former students. She has been teaching in poor neighborhoods and a significant percentage of the parents are illiterate. While these parent's are very happy with my wife and they request all their children (and cousins etc) have her, using their feedback to rate my wife as a teacher would be seriously flawed as they really don't have the ability to judge if the teaching is effective at all.

Perhaps we should ask the students and student parents? After all the parents are paying the bill (even when it's indirect via taxes)

ultimately tie teachers' hands and force them to teach from only state-approved curricula

But that is the point! If the union makes it impossible to fire a bad teacher, you must try to limit the damage that teacher can do, hence the set curriculum. If the union makes it impossible to single out bad teachers for this kind of prescription, then you must do it to ALL teachers.

Get the union out of the picture and the situation becomes self-correcting.

You're too focused on bureaucratic nonsense (unions vs. administration vs. state vs. local vs. federal government.)

In fact, it seems like you're just taking it as a given that unions are the problem and everything will be made better if you take away the teachers' first amendment rights to assembly and petition for redress. Which has nothing to do with teaching students, which is the problem at hand.