Gates Foundation study: We’ve figured out what makes a good teacher
washingtonpost.com
washingtonpost.com
In middle school, the last three weeks of each semester were spent watching movies (the same two or three movies over and over). In high school, I was taught that the big bang theory was when the continents divided. I was taught that the most direct route between two points on the globe is through the north pole. My Spanish and woodworking teachers taught math because they knew that the math teachers weren't doing it.
When you're talking about schools like the ones I went to (which weren't nearly as bad as many other schools in America) I think that test scores are a decent indicator of the education a student is receiving. You're measuring things like "can this kid read English" and "can they multiply two numbers together." Standardized tests are certainly capable of measuring things like that. Testing probably breaks down for measuring intelligence/education as the level of education improves, but I encourage everyone to consider just how dysfunctional many schools are before ignoring testing entirely.
The rest of the student body was made up of poor, mostly black students who came from families where education wasn't a priority. Every single year I was in high school the state tried to revoke our accreditation and we just barely passed a state audit to keep it. I was a math tutor, and it was common for students to come in who couldn't do basic addition (in high school).
To make matters more confusing, the gap between the good students and the bad was almost entirely caused by economics (wealthier parents resulted in smarter kids, as is normally the case) but because the economic divide was almost identical to the racial divide, many of the educational issues were easily conflated with racial issues. This led to the (mostly black) teachers and administration demonstrating favoritism toward white students, and it also created tremendous social pressure on black students not to try to succeed academically (otherwise you're an oreo).
I wouldn't trade my experience there for anything. I received a pretty good education, and I experienced a part of America that more privileged kids should experience, but most of the students at my school were definitely being left behind.
As for parents not getting involved, I'm not sure I ever told my parents about that stuff because I thought it was normal. Middle school was by far the worst part of the district, and a few years ago my brother and I were telling my parents stories about how about once per week there were school-wide riots in the lunch room that the FAs (bouncers) couldn't contain and we could do basically whatever we wanted for 30 minutes because of the chaos. It turns out my parents had no idea this was happening (my mom said she wouldn't have let me go back to that school if she'd known). We never told any parents because we thought every school had riots like that.
I'll add to this since I think these types of stories occurs more often than people think. I went to Baker High School in Mobile, Alabama (that might be your first clue).
In the 8th grade I took a required course called Earth Science, also of course taught by a coach. I recall that the last question of one of the exams was "The Universe was created by _____________". Odd. I thought this was a fairly complex answer for a simple fill in the blank question, especially for tests which were typically always multiple choice or fill in the blank. So I just assumed that the empty space below it (since it was the last question) was for writing a short answer.
So, I did my best to mention the Big Bang Theory and associated information about red shifts and such for scientific evidence. In the end, my answer was about a paragraph long, and I felt really good about my answer.
I did get full credit, but I noticed that other students also got full credit but wrote only in the blank itself. So I checked with the teacher for how I could have been more concise. Turns out that the correct answer that he was looking for was "God".
I guess the question was a simple fill in the blank after all.
This is the most dramatic instance I experienced, but in Honors English we spent at least a week on the Bible, but never any other religious texts. Conveniently, at the same time I had read some vampire fiction by Anne Rice and concluded that it's possible that Satan is actually the hero of the Adam & Eve story. It turns out, however, that this is not a good opinion to express when trying to do well in English in a conservative public state school.
As to Earth Science, I was not penalized for my response; had it been, this would have been clearly objectionable. On the other hand, at the time I did feel that it was unfair in that one could give what I considered a cop-out answer for full credit. It may be a theologically true response, but it does not give us any scientific insight and understanding into our Universe.
Given the two examples above, I'm not really sure. It can certainly be uncomfortable for someone who is brought up as an atheist, but I'm not sure it's outright illegal. And this is where the avenue for standardized testing comes in -- it's not necessarily to demonstrate student learning but perhaps rather to better identify egregious issues in school districts.
I'm quite sure it is illegal according to federal law. The question is whether state law trumps this. Because if it does, no doubt Alabama has passed a law making it legal to teach creationism in schools.
Which is fine because if the subject matter actually matters, there will be ample chances to relearn it later. Having teachers teach stuff that is wrong is only a big problem if you think teachers need to be infallible. But the fact that teachers can be wrong seems to me one of the more important true things kids learn in school.
The best predictors of future success are self-control, emotional intelligence and a grasp of functional algebra (which I believe, but don't have evidence to support, is because of the importance of computational and systems-level thinking.) Only one of those, self-control, is even indirectly measured by current standardized tests.
True, but there are also many families that can afford books but don't buy them, instead spending money on other things. Not having books in the house has been shown to correlate with decreased academic success.[1] I'm not saying this isn't a wealth issue, but that probably isn't the only dimension.
[1] http://www.sciencedirect.com/science/article/pii/S0276562410...
Maybe I misunderstand you, but I fail to see how multiplication or reading are not useful skills. Good reading comprehension will help you navigate life in general, which is something lots of people struggle with. Similar for multiplication.
Even if someone is not very intelligent, if he's mastered a basic skill like that, he'll find lots of places to apply it. That's why there's so much emphasis in school on them.
Also, my school emphasized self-control, but how do you learn emotional intelligence?
It will take further work to associate learning such skills with the positive outcomes associated with the skills themselves and figure out what the best ways to teach the skills are, but current research is promising.
I don't think this is true. Those who can't afford books can goto the public library and get them.
We toss lots of books in the trash in America, not because we hate books, but because we make so many of them and then need to make room for the new books.
"Able to afford books" was once a status symbol. It's not any more.
See also: http://www.npr.org/blogs/monkeysee/2011/10/12/141265066/hard...
If a student does at least moderately well on standardized tests, there's a decent chance they'll do at least moderately well later in life.
If most of a teacher's students flunk completely on the standardized tests, the teacher probably isn't doing a very good job.
They do provide weightings based on optimal predictions for maximum accuracy with predictions for state tests scores, although those weights appear to assume a simple linear correlation. Even then, the data suggests that in class observations are almost a complete waste of time (single digit weightings), and that student surveys decline in value as the kids mature (no shocker: with younger kids, how much they like the teacher has a much more profound impact on their success at learning). In all cases the kids seem to be a more reliable indicator than the observations of professionals! This may just be an indication that assessments based on more frequent observations are more accurate, but the only way we can really get a sense of that is if they also had peer observations from team teaching environments... which are completely absent.
That's not the worst of it though. The worst is the rest of the data. It is based on some fairly arbitrary presupposed models for combining these different assessments. No breakdown of the individual components of these assessments (other than "English" & "Math") and no attempt to discern a model from first principles.
There are some other factors as well. Aside from some outliers, the accuracy of predictions seems kind of suspect. Most of the outcomes are within +/-0.05 standard deviations, and a majority of them are within what looks like +/-0.025 standard deviations. These are pretty minuscule variances (0.25 was supposed to represent a year of schooling, so 0.025 represents maybe 15 school days worth of progress at best), and it is not hard to see that without some of those outliers, the regression line doesn't look so pretty (particularly for English). I also didn't see any metrics on the variance of the data within a classroom, which I suspect makes any predictive value look limited at best.
Ugh. I really hoped that Gates would make sure there was more statistical rigor with an analysis like this. :-( I hate to say it, but the conclusions of the study are not well supported by at least what is in the final research report.
http://marginalrevolution.com/marginalrevolution/2010/09/the...
Good on him.
>> The teachers who seemed to be effective were, in fact, able to repeat those successes with different students in different years, the researchers found. Their students not only scored well on standardized exams but also were able to handle more complicated tests of their conceptual math knowledge and reading and writing abilities.
To everyone who keeps saying "standardized tests are bad", please cut it out. A particular code benchmark may be useless, but benchmarking overall is good. Existing standardized tests may be bad, but in general, having some way to make objective measurements and adjust accordingly is the basis of all human improvement.
And yes, we should reward good performance, if we can determine what good performance is. How else do you expect to get more of it? It is incredibly difficult to make incentives that actually do what you want rather than encourage gaming of the system, but that doesn't mean we shouldn't try.
In short, let's stop debating whether we should measure and incentivize, and start debating how.
I did pass their tests, but did I get a good education? Frankly, no, I don't think I did. I mean statistically I got a great education, but it seems we have forgotten why we even educate to begin with...
Do we educate so people look smart on some random league table? Or do we educate so people are actually more well rounded, have a better grasp of the world around them, and do better as adults?
I think we're all for this type of metric. The problem is that no one has yet been able to show a teacher's ability to achieve that.
So you're against testing. What metrics are you for? Do we at least agree that without any sort of metric to judge a good teacher vs a bad teacher that the whole educational system suffers tremendously?
When tests become a core part of how you judge someone's worth as an educator that educator is going to put all of their efforts into better test scores.
The question is do test scores mirror the quality of education or do test scores just tell you someone's potential at performing well on a test?
Testing is just one part of a three-pronged approach that the foundation is recommending. The other prongs involve classroom evaluations and student feedback.
Students are not a commodity. What works well for one person does not necessarily work well for another. Without focusing on the individual, you are always going to leave a group needlessly behind.
There at least a couple obvious groups qualified to rate educational quality: teachers and students. If they happen to choose a standardized test to evaluate the education, then fine. Otherwise, I don't see standardized tests as obviously virtuous.
Universities do use standardized tests: they use the SATs and the ACTs, the GREs, LSATs, MCATs, Miller Analogies. They use AP exams to determine who gets college credit for courses the school never administered. They use their own standardized entrance exams for placement.
Because the government said it was illegal. (I'm simplifying, but look up Griggs v Duke.)
However, an employer can look at your college, and your college definitely can use your standardized test scores. So employers do use standardized tests, it just has to pass through a socially accepted filter first.
1. How wealthy, poor, educated, or uneducated a child's parents are.
2. How many resources the school and teacher are given.
3. Whether a child's parents are abusing them at home.
4. Whether a child is witnessing violence, hopelessness, despair in their neighborhood.
5. The attitude of a child's family and friends towards education.
6. Whether a child's parents are active in their child's education and in a PTA.
Study after study has shown that these factors greatly influence a child's attention and attitude to school work, their attendance, and ultimately their performance at school.
No child's performance on a standardized test, and no evaluation of a teacher's performance is going to pick up on the above factors, yet they can critically impact how well the child ultimately learns and what the child learns.
Standardized testing of a child that is unfortunate enough to live in a situation where the above factors are against them will likely notice a poorly performing student, one who gradually does worse and worse on the tests until they finally flunk out.
Then, instead of getting extra aid, the funding to the school that has a lot of such children will be cut (thanks to programs like No Child Left Behind), and their teachers penalized.
These factors, which are strongly associated with poverty, are things a given teacher is in no position to influence, but which society as a whole could address. If it has the will.
But yet I'm judged on the small slice of that overall pie that I can control.
What this study indicates is that there are teachers who can reproducibly create success levels given the control of the situation that they do have.
How is it any different than the rest of the working world where you can control some things and you can't control others yet your job is to do the best you can with what you can control?
Whether standardized tests have a role in answering that question or in evaluating teacher performance is hotly debated right now.
But the purpose of my original post was to draw attention to a different question: How can society as a whole ensure that every child gets the best education possible?
In schools that have no problem attracting resources, great teachers, and children with mostly fortunate backgrounds, perhaps this second question can be answered by asking the first: how can we improve our teachers?
But in schools that lack resources, whose students live in abuse, poverty, and despair, what even an exceptional teacher can do will be radically limited, and society would probably be better off focusing on how to eradicate poverty -- the real cause behind the failure of too much education in this country.
Unfortunately, seriously tackling the issue of poverty is much harder (and even contrary to some dominant American ideologies) than simply demanding more standardized tests be given and blaming teachers and their unions.
Step one, define "best education possible". Step two, learn to measure whether or not a child is getting one (or the closest you can come to it). Step three, maximize some aggregate or regionalized version of your measurements through restructuring of school systems, rewards for good schools and teachers, etc.
The study in this article is just gets us to step two. It looks like you're wanting to discuss step three. That's a different can of worms.
I know that the funding the schools get is blamed for a lot of their problems, and I understand where people who talk about how hard it is to teach kids with behavioral issues and histories of abuse are coming from.
I sympathize, but I also think that's a cop-out. I had bad teachers in both places. In Winthrop, I had horrible teachers. They took away advanced classes because an outside contractor thought it created elitist attitudes and putting smart kids in the regular classes would boost overall performance.
I had science teachers that just had us play with legos all day and gave A's to all the basketball players. I had math teachers that would lose assignments and blame the students. These were not well-meaning teachers that just had it hard trying to teach these poor kids. These were people that shouldn't have been teaching at all, but could not be replaced.
My parents actually went up to the school a few times to try and talk to the administration and teachers about what was going on. A lot of people did. They got push-back and were asked not to meddle.
There were 2 great teachers there and that's why I didn't completely screw up my life at that point. I've spoken to a lot of people with similar experiences; 1 great teacher that pushed them to be successful. That's why it's so maddening to me when people say that it's not the teacher's fault, that they can't do anything about it. I know that's not entirely what your intent with your post was, but focusing on external environmental issues and bemoaning the poverty of the student body can come across that way.
While poverty is a major issue, and teaching is hard, I don't think you can solve the problems by just changing the student's home environment. You have to change the teachers as well.
Lots of jobs are hard. Doctors have to deal with unruly patients who refuse to change their diet or exercise. Don't they get blamed when their patients don't improve?
So the argument is that since everyone sucks at measuring things that any arguments about how a particular system of measurement is ineffective should be discarded?
You read something into my argument that I didn't put there, so I'm going to disregard it as a straw man.
I think we can all agree that judging people based on things that they have no control over is not an effective way to evaluate them. This happens in education, as the parent comment outlined. This happens in engineering, as you outlined. It does not follow that teachers should be evaluated in this way, or that a discussion about the problems with this method should not take place.
The study attempted to control for things the teacher's couldn't control. They're actually being differentially judged on the parts of education that they can control.
These lines from the article fly in the face of your entire argument: "They also ranked the teachers using a statistical model known as value-added modeling, which calculates how much an educator has helped students learn based on their academic performance over time".
You would need to address them directly in order to mount a credible counter-argument.
Why not? Government run education is pathetic. Incentives are all out of whack, there's no competition between teachers or schools that can affect the status quo.
Teachers could use a huge dose of whatever it is that the more competitive business world has to deal with.
Teacher competition, I think, would reduce collaboration. Why give your competitor your secrets to success?
Incentive pay? The problem under this idea is that new teachers are often given the most difficult and unruly of students while more experienced teachers get honors classes. I find this odd because the teachers will the best "tools" for dealing with difficult problems are moved away from those problems.
Then again, the problem is that with any idea for reform, there is a list of reasons why it should not or would not work.
I'm a fan of letting students who are not interested in the academics of school work towards trade school and let those who are academically inclined to work towards university. However, a retired college physics professor friend of mine does not like this solution because he would have been tracked towards trade school and never discovered his love for physics.
Just like you'll see recommended on HN, if you're going to fail, fail early and get on with something that has a chance of working. Our government funded education system is a long painful failure that established forces prevent from being scrapped and reinvented.
Imagine being a situation where not only did you not have any control over which engineers worked on your team - or even whether they were engineers at all, or even basically literate. Now imagine that your whole industry pretty much worked liked that and that changing from one company to another (which most likely worked the same way) always meant moving cities.
Incidentally, what they did in this study is to measure the value-addition of each teacher, not their raw test scores. Presumably you're not evaluated based on things you have absolutely no control over, as you said - you get judged based on the slice of the pie that you do control.
Hell, I've only ever fired one engineer in a 20+ year career, and that was because he was the equivalent of a violent student whom you would expel and send to juvi.
Why is it so hard to deal with the information as presented in the study. Fine, question the details or methods of the study. Offer counter evidence. The study presented evidence that good teachers can have a noticeable and measurable level of control on the output of their students. Why is dealing with evidence not the central part of this discussion?
This was an Algebra class, and the students had to be graded on their Algebra skills and judged on their Algebra proficiency tests. No homework would be completed, and no recourse could be given for lack of homework. 50% of these 9th grade students will not make it graduation. 3% will make it to a four year institution. Less will graduate from there.
How is this different than working with professional engineers? Ones who could be fired or in some other way worked with? How is this different than working code where a solution exists? How is this different than working with other people who are looking for a solution as well?
A better comparison would be to how you judge a prison guard in relation to recidivism rate.
As an aside, it occurs to me that those teachers who are most effective in underprivileged schools may be totally different than those in wealthier schools.
Whenever teacher/school issues come up, it is important to remember that far beyond anything a school or teacher can do, the single-most important factor in a child's education (or their life in general!) is their PARENTS. If a kid has parents who value education, discipline their kids and make time for learning at home, that kid can do just fine no matter what kind of conditions they experience at school. Conversely, it takes a Stand and Deliver-style superteacher to be able to coax success out of a kid growing up in a home where education is not valued and productive personal habits are not modeled.
The biggest problem with education? There are far, far too many parents out there who have absolutely no business raising children at all. We require tests for people to operate a jet ski, but when it comes to absolute power and responsibility for a human child, anybody can do it! That's your right! Don't worry about saddling the rest of society with your child who doesn't stand a snowball's chance in hell of ending up well-adjusted, educated or successful. It's YOUR kid.
I understand that people who are struggling with extreme poverty are often too caught up with bare survival to find the time to focus on their children's educational needs. That is really awful and sad. It is especially sad for their children.
The fact is, as long as society is full of self-centered, thoughtless parents who either fail to understand the basic components of child-rearing or are too busy surviving to care, we will be plagued with students who are going to struggle no matter how great of an educational system they are placed in.
That being said, perhaps the only way to raise a generation of education-valuing, quality parents is to do what we can to create them. So maybe it is best after all to focus on what CAN be done - effective teaching methods for today's teachers. Hopefully a much-improved school system can help raise the prospects of children whose parents are doing nothing to give them a hand up. And then they in turn can become the kinds of nurturing, education-valuing parents that their own children deserve.
Unless the student's parents started abusing them in the year the teacher is being evaluated, it is already accounted for. VAM measures deltas, not absolute levels.
However, see this post for a strong critique of one of the key claims of the MET study:
http://garyrubinstein.teachforus.org/2013/01/09/the-50-milli...
Namely, in the scatterplot correlating predicted effectiveness and actual effectiveness, they pooled teachers into 5% groups. This naturally reduces the variance, yielding a much straighter line of points than would an ordinary scatter plot. This is not, to my knowledge, an accepted or ordinary technique.
The 5% groupings doesn't evaluate the metric. It just means that if they plan to use this actual study as a guideline, they should use the same technique in grouping teachers.
What happens when you find the bad teachers? Fire them and hope the fresh-faced college grads do a better job?
So much of this debate on teachers is leans towards the carrot or the stick. Wanting to reward the best teachers because it makes you feel like you're helping is fine, but don't fool yourself into thinking it'll hugely improve instruction. Teachers aren't going to suddenly go, "Oh, wait, you'll pay me more if I'm a good teacher? I'm going to start trying!" (One or two might, but those outliers exist in any field, and are hardly the issue.)
In every district I've taught, very little time is given for teachers of the same subject to collaborate, observe each other, and improve their own teaching. Every year, we have to get 20 hours of professional development. Tell me, are 6 minutes and 20 seconds of development per teaching day going to improve instruction?
at which the latest project report can be found. I've got some reading to do before I think about what else to add to the discussion already underway here. I get the impression that one observation that has motivated much of this research is how persons educated overseas have fared in United States society after immigrating here in early adulthood. My own experience living in another country (twice, in two different three-year stays) does much to prompt my interest in education reform, which is what drew me here to Hacker News.
You can find a measure that correlates to performance, but once you use it to actually evaluate teacher, teacher will adapt and the measure will lost all relevance.
Rely on test scores? Teach the test, not the material. Rely on student evaluations? Bribe them with easy material and good grades. Rely on principal evaluation? Get on its good side.
For example the budget of this study came out to about $15000 per teacher. What if instead we spent $5000 of that for professional development / training for the teacher, $5000 for a classroom set of chromebooks, and gave the teacher a $5000 raise. That alone would cause some significant improvements, I believe (not that that is ALL that is needed, and there are so many bigger issues that need to be tackled as someone else mentioned, such as poverty and inequity).
But when we talk about "good" teaching and "bad" teaching, we are treating it like it is a simple craft that is mastered or not mastered. Teaching is like engineering, it involves design, and design is about making things better (see this prezi on learning design: http://prezi.com/b44jwdgvs8nl/olds-mooc-introduction/ )
The goal was to identify what actually works, which is something you need to know before improving the situation. Paying a bad teacher more will not make the education provided any better (despite whatever the teacher's union says).
Dewey believed that the goal of education should be to produce good citizens. Systems with metrics based upon the model of student as consumer are more likely to produce outcomes biased toward producing consumers.
The entire set of good teachers includes those who inspire a lifetime of learning. It includes those who teach kindness, sportsmanship, and good humor. It includes those who teach subjects for which there are few students and in which standardized tests make little sense such as art and shop.
Maybe the study can point us in the right direction. But few districts will bring outside evaluators from Cambridge's famed institutions. Few districts will fund ETS analysis or purchase their tools.
What we are likely to get are processes localized to the political reality on the ground.
[0]http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html...
In fairness, I think I heard they realized something like "what works in america doesn't work everywhere" (bit of a cop out to save face) and moved toward more sustainable organic methods, but I'm not completely sure.
All I know is, if its that hard for them to figure out the most practical way to grow a plant, we should probably take their advice on growing the minds of our children with a grain of salt.
On the biodiversity thing, viewing your website leads me to believe you appreciate nature, so I'm just scratching my head that you haven't made the connection. Ohh well, downvote me all you want buddy. You are entitled to your opinion also.
Great. Where's your evidence?
My own evidence (experience going through school and generalization of how ability works in most professions) tells me that the good teachers in my life helped me to learn the information better and more deeply. My memories of those classrooms are of the subject matter and my desire to achieve in that teacher's environment. The crappy ones were noticeably so in that my memories of those classrooms was of misbehaving with my friends, having no desire to help the teacher teach me, and cramming to learn material for tests because I hadn't absorbed anything beforehand.
1: If we are looking at a teacher who teaches a course with a pre-requisite, the grouping of students becomes more similar (ie, a trigonometry teacher is likely to have students both motivated to do well and to have passed the requisite lower level math classes, and their success is less surprising to me).
A study comes out from the Gates Foundation that tries to tackle some of the education problems head on. It produces some statistically interesting results, but here on HN, I keep reading knee-jerk responses about "standardized testing" from people who apparently didn't even read the attached article.
How are we going to solve these problems if no one even bothers to understand our options?
Regarding your point - I agree. Any solution would need to account differences in what you referenced as a "variance in 'quality' of student".
Brilliant. Well, yes, that is the best way to measure a teacher's efficacy, if your goal is to teach students to jump through hoops and pass standardised tests rather than to think.
The best teacher I ever had was my grandmother - she taught me the most important lessons I'd ever learn, such as:
"Observe the world around you."
"Reach your own conclusions."
"What was fact yesterday may be fiction tomorrow."
Tests are a distraction from learning.
You're tested throughout your working life. Almost every job interview is a test. Giving a customer presentation is a test. Finding bugs in your software for a release date is a test. Memorizing that big Cheesecake Factory menu so you can wait tables is a test. Getting your General Contractor's license is a test. Getting certified as a Pilot requires passing a test.
There should be more to education besides testing, but being tested is a part of life and it's easy to measure success - that combination means that it's a useful tool for education.
The others are not good examples because they are useful in-and-of themselves. Giving good presentations and finding bugs are useful because they are useful, not because they predict something else.
Pilot's licenses, like driver's licenses are based on tests where you more or less do the thing that you're being tested on. The test and the thing that it's supposed to measure are very similar and we can therefore use it to predict performance. In cases like that "teaching to the test" doesn't exist because practicing for the activity and for the test is the same thing.
I'm not denying that educational testing is useful to quantify progress but such tests are often different enough from the underlying activity they're intended to measure that effort spent preparing for the test has little other purpose.
Giving a presentation isn't a test, it's a pleasure. You're not being tested. You're just presenting facts for someone to make a decision. There's not a right or wrong. If there is, you're in the wrong room.
QA is not a test. It consists of tests, sure, but those are for the software, not you. It's a task, and one with a purpose.
Memorising a menu is not a test. It's memorising a menu, for a purpose. If you were memorising a chapter of Crime and Punishment verbatim to get a job at Cheesecake Factory, that's a test, and an arbitrary one.
Getting a license, in the sense that you're tested, again, is different, as those are vocational tests which test your practical skills for a specific application.
Being tested is only part of life if you choose to let other people set your challenges. This is what the educational teaches you to do, and accept as normal.
Life isn't about passing tests, it's about learning.
"testing students more often to improve test scores is similar to a rancher weighing is cows more often to increase their weight."