Hacker News Reader

Comment by mlyle | Hacker News Reader

I think it's pretty damning, actually. Spending most of your time circumventing the test and rote repetition undoes any learning of the real subject.

I taught AP Microeconomics this past year. About a third of my time was spent teaching about different ways that the AP grading standards are picky and ways to exploit the test, because this is what is most important for the outcome they're worried about-- scoring a 5 on the exam. In turn, it didn't leave a lot of time for deeper discussions, simulations, etc, engaging with microeconomics.

View on HN

How my AP Econ (Macro+Micro) teacher tackled that was spend the majority of the the time teaching the Samuelson and Mankiw books and then spend 1 month having everyone practice College Board style tests.

Tbf, in our HS AP Econ was gatekept for 12th graders, and most of us had at least 6-7 AP tests done already so we had experience with the exams already.

I had mostly seniors with lots of AP experience too.

AP Macro+Micro can work a lot better, because there's a few weeks of overlapping material. So it's the difference between fitting 45 hours of instructional material into 50 hours vs. fitting 80-85 into 100.

The tight fit annoyed me, as did the strictly graded nature of the course.

Two areas of particular annoyance:

- I had a cohort of students who had entirely been through AP Calc, but fitting into the explanations expected for HS meant making somewhat bogus algebra based explanations (the slope of this line is double the other one, just because!!). A student who gave a calc-based explanation on the test would probably not get the points (frankly, a lot of the readers would not understand it).

- About a third of the students had founded a "Game Theory" club on their own and were off in the weeds exploring game theory. I'd have loved to have taken a couple extra classes on the topic since they were already so interested and engaged, but I had no margin. I was forced to give the very simple 2 person payoff matrix form and leave the topic there. Again, I spent more time encouraging my students to not think too fancy.

This low key sounds like the environment in my high school XD.

> had a cohort of students who had entirely been through AP Calc, but fitting into the explanations expected for HS meant making somewhat bogus algebra based explanations

Yep. I remember that. We had the same problem as well in our class so our teacher spent that month prior to the exams teaching us Econ the Wrong/College Board Way.

> I was forced to give the very simple 2 person payoff matrix form and leave the topic there

Yep, I remember that too. A bunch of us were CS/Applied Math bound so we ended up expanding further on Optimization Theory and Game Theory via competitive math exams or working with our supportive AP Calc and E&M teachers who taught us how to hone knowledge in both concepts using math.

Odds are high we are talking past each other. Teaching the rules of the game is the same as teaching grading rubrics in many ways. And yes, learning the nature of rubrics is very important for students to know.

Now, can you devolve to completely degenerate states where they are only playing the "meta topic" and not learning the topic? I mean, yeah? I'd wager the folks that can complete the meta topic still know more than the folks that can't even do that. You can also devolve into stagnation by thinking you can reduce all learning to "first principals." Such that I just don't see that as damning on its own.

Back to my sports comparison, the technicalities and fundamentals of a sport are important. But you also do well to teach kids to get out there and play the game, even with loose rule enforcement and heavy help. Bumpers in a bowling match, as it were.

> Teaching the rules of the game is the same as teaching grading rubrics in many ways.

I'm a big fan of teaching rubrics and even test-taking skills.

But that's a far different thing from eviscerating your subject in order to teach points tests are likely to hit in isolation, which is happening a whole lot, especially when teachers are evaluated heavily on test outcomes.

Right, this is why I'm guessing we are talking past each other. I fully believe some places got degenerate and did it poorly. I'm not convinced every "teaching to the test" case our there is that, though. Often folks would whine about teaching to the tests for teaching how tests are scored.

I'd love to see a large study on this. And now that I have kids in schools, I'm very interested in ways I can help them learn. At the same time I also want them to do well with grades. Knowing that is often two separate things.

For an example of how it can go wrong: I know for a fact that the competitive math class I teach makes a couple of standardized test measures go down-- especially "procedural knowledge". I know this because I've compared changes in performance to similar control students not taking the courses.

I also know that it has a big positive impact on performance overall. And our students do pretty well in comp and a lot reach state championships.

If I were strictly evaluated on standardized test outcomes, I would be asked to figure out how to pull those procedural knowledge scores up.

Right, and I am not really arguing against you on this. For an example of places where not teaching to the test can go wrong, though, I need only look at some extended family that were only taught evolution, as it would be on some tests.

Anecdotes being what they are, I'm interested in knowing if there is a large study on this. I'm also very open to thoughts on how to fix things. Knowing that it typically comes around to better tests...

Edit: To be clear, my evolution example was supposed to be that they were only taught it because it would be on the tests. Not that they were only taught evolution.

> if there is a large study on this

I'm not sure what "this" is, though. The question isn't well-posed, so I'm not sure what you would study.

Absolutely there are many public school teachers that report that they are spending a lot of time teaching "to the test" rather than what they consider important. And this is a case of misaligned incentives. Overall mastery is important for the student's outcome going forward; the outcome of this, say, 5th grade standardized test doesn't affect their educational trajectory at all.

But the outcome of that test is very important to the school's perception and the teacher's career.

We'd all love better tests that really measure what we're trying to inculcate in students. But we also want tests that are easy to grade and very quantitative, which is in opposition to the other goal.

"This" would be how many places are harmed by "teaching to the tests."

For example, you are asserting that "there are many public school teachers that report that they are spending a lot of time.." but... where is this tabulated? Are there biases mitigated in the self reporting side of what you are talking about?

I don't want to be dismissive of the idea that it is harmful. I have seen very little compelling evidence that it is as harmful as it is asserted. Both as a student and as a parent. Is akin to employees that are upset about long term compromises for short term gain. Yes, it happens. All too often, short term survival is the best thing for long term survival. (With the ack that it is until it isn't.)

Which is why I would love to see data and studies on this. Indeed, if "teaching to the test" is the concern, how do I then interpret declining test scores? Failing to teach either to the test or the subject?

> where is this tabulated?

How would one tabulate this or conduct such a study? Usually the main outcome measure you're looking at is something like the test.

> if "teaching to the test" is the concern, how do I then interpret declining test scores?

Exactly what you mentioned before and dismissed: short term gain is not necessarily long term gain.

There's not enough high quality educational research. But one area we have strong research is on what kinds of learning are durable. The National Training Laboratory Institute has done a whole lot of research on what results in retention at 6 months, and the things you do for immediate test performance (rote practice, mnemonics, tricks to answer specific subject-specific questions, etc) are the least durable things.

Test focused classrooms spend a whole lot of time on this stuff, and less time on things that work to create durable understanding (think-pair-share, low intensity practice built around spaced repetition, mastery-based methods, etc).

Note that I am not a career teacher. I am a retired entrepreneur who now happens to teach. I have extraordinary programs that get extraordinary results-- despite being a small school, we're routinely having multiple students reaching state level championships in things I teach. Under test-focused pressure I could not achieve these things. Similarly, if I was spending a huge portion of my energy on classroom management and behavioral norms, I couldn't do them either.

(Of course, on most test measures my students do pretty well, too).

Indeed, I've written here: https://news.ycombinator.com/item?id=36423142 (and in the grandchild comment) about my experience teaching a very heavily test-focused course. I know I could have given my students a better and more durable knowledge of economics with less of the test-focused pressure.

My assertion, then, would be to point that "teaching to the tests" concerns started and continued through a time that tests scores have decreased. If they really are teaching to the tests, they are doing a bad job at that, as well. :D

I didn't dismiss the idea that short term gain can be a long term loss. I question if that is a panacea of a claim. Specifically, all too often, I see it pushed by fellow programmers when they are pushing for a harder solution and have stalled out on it. I have been said programmer doing that, before. It is best exemplified in the nonsense talk people have about how "worse is better." Antifragility appeals to me. Total ordering of ideas, required for "worse is better," do not.

Worse, I've seen students that did a lot of these supposed better methods fall flat on their faces when in a new environment. And then it falls back to an obnoxious "true scotsman" debate about how they must not have been in these superior settings, as otherwise they would have succeeded.

To that end, I would challenge that you could have really given them more durable knowledge of anything, right off. If only because I have yet to see anyone succeed at giving durable knowledge of anything that wasn't flat facts. I'd love to be shown how that is wrong, so please don't let my skepticism keep you from doing what you do. I'll go further and say that it is very likely that my skepticism is unhealthy at some levels.

And again, I am interested in seeing more exploration of all of these ideas. As a parent, I'd love for my kids to get good at all they want to do. Rote practice and simple following of recipes and such is far better at that than it often gets stated. (And a big "f you" to "sight words." I can't believe how misguided that is and how it set back one of my kids in reading.)

> would be to point that "teaching to the tests" concerns started and continued through a time that tests scores have decreased

This is false. The concerns of teaching to the tests, from my memory, peaked around 2005. No Child Left Behind was a major driver of these concerns, passed in 2001. Test scores peaked in 2012. This is exactly what you would expect if "teaching to the test" yielded a short term benefit but longer term harms, and was increasing during this interval.

> Worse, I've seen students that did a lot of these supposed better methods fall flat on their faces when in a new environment.

NTL did pretty simple experiments: teach something various ways, and see what ones resulted in best performance 6mos and 12mos later.

Having students themselves pair off, teach, and explain ideas is the most durable thing we know. It requires active engagement with the material in order to reformulate it to help someone else through understanding. It requires being able to do this while multitasking and attending to social cues. And it can be a powerful motivator for students to fully understand before undertaking explanation.

> (And a big "f you" to "sight words."

I can't figure out what you're saying. "sight words" are a classic example of a hyper-focused rote memorization practice.

I'm not saying rote is worthless. From my perspective, a perfect math class equally emphasizes intuition, rote practice, and rigorous explanation. What we tend to get, instead, is that once a student falls at all behind, they get 80+% rote practice. Worse, this practice is often subtly faulty and on topics that the student doesn't have the higher level explanations necessary to monitor their own performance.

Teaching to the tests never stopped being a concern from the circles I can remember. No Child Left Behind is an interesting fence post on this, though. "Sold a Story" was rather effective at showing the major opposition to all things Bush in that legislation and how it backfired heavily for reading. As so many opponents of that act railed against how it was pushing for reading to be tested, and how we are still struggling from that.

That said, I do not have hard data here. Happy to be shown I was wrong that that peaked about that time.

For teaching/learning, I confess I'm more partial to https://www.scientificamerican.com/article/getting-it-wrong/. I'm curious to know more on studies like this, so I'll be looking up some of the NTL studies you are referencing. Would be delighted if you have a good reference on them.

My "f you" to sight words is just hearkening back to the "Sold a Story" thing. So many of the "best practices" for teaching reading from teachers centered around "sight words" and such. Which, I suppose you can frame that to teaching to a test. But I think it is more teaching the wrong things. And flat out not teaching any technique to reading.

Which, that last is fully to your point. I don't want techniques ignored or passed over. So if there is large scale evidence that that is happening, I'm very interested in the evidence.

> That said, I do not have hard data here. Happy to be shown I was wrong that that peaked about that time.

I'm consistently confused about what you're saying. Peaked during what time? You've said it peaked -after- test scores started declining. I said it peaked before tests started declining in 2012.

It indeed peaked before 2012: https://trends.google.com/trends/explore?date=all&geo=US&q=%... You'll find most of the most critical articles in 2001-2004 if you search past media results, too. There's been a bit of a recent resurgence.

> For teaching/learning, I confess I'm more partial to https://www.scientificamerican.com/article/getting-it-wrong/.

I have no problem with hard tests given to students as a learning tool, which is what your article advocates for. Indeed, I use difficult pre-tests a lot, and my target % correct on tests that I give to students is a 60% average, because this gives me the most statistical power to measure knowledge and students performance. Because a very large proportion of my students are above 80% in national norms, giving them a test that they'll miss 40% of available points is a quite difficult one.

I am not a fan of tests used to measure educators: not because measuring educators isn't useful, but because I believe it incentivizes the wrong behaviors from educators. I consider myself fortunate to not be managed in this way. The time when I got close to this (teaching an AP class) is an experience that I don't really care to repeat, even though my students did well and and my class was one of the most popular at the school. I felt like I did not have enough freedom to do well (discussed above).

Apologies, went to sleep.

I don't actually see a defined peak in that trend chart. Such that I'm not clear what point that is making, all told. Though, I also didn't claim that worries on "teaching to the test" had peaked. That is your claim. If it did indeed peak during that timeframe, I'd be game to see explorations of that. If the linked chart is all we have, that looks more like noise and hints that the regional variation in how hot of a topic it is dominates. Which could explain why you noticed it drop off in your circles, but I did not.

I also fully suspect we would be in more violent agreement on these topics than this thread looks. :D I also do not like scoring teachers in that way, necessarily. I don't have a good alternative, but I fully agree that it screws up incentives and is the direct reason we get the degenerate cases. Cheating, in particular, being far worse than teaching to the test.

Just to clarify what I was saying:

You claimed they -started during declining test scores-: "My assertion, then, would be to point that "teaching to the tests" concerns started and continued through a time that tests scores have decrease"

Meanwhile, they clearly were very big from 2001-2005 (and according to linked chart, have decreased in intensity since that time). And test scores started to decline around 2012. That is why I mentioned that they had peaked around 2005, because that was well before the period of declining test scores.

Ah, I think I see the main contention.

Most of the time I hear folks talking about "teaching to the tests" they are upset that they are just getting kids so that they can pass the tests. Indeed, even one of your complaints was that you had to compromise your teaching so that they could score better. My point is that if that is the concern, they are failing even at that. Test scores are going down, such that most of the arguments that they learned the tests, but not life skills, are moot. They aren't even passing the tests.

That is, I didn't mean to claim that they simply started during declining scores. I meant more that the scores are declining now. If the concern is that they are "teaching to the tests", my bigger concern is why are they so bad at that, even?

For timing, the assessments seem to be that they have been declining steadily since 2012. If your claim is that the "teach to the tests peaked in 200x" and that that is why tests are failing, that should have started reversing with kids this year? (I assume the assertion is there would be a 7ish year lag on kindergarten kids getting to age 13, to be reflected in this story. Obviously not a full step function, but I don't have a solid model on that. Love it if you have one.)

To that end, my assertion would be that if they were at least able to pass the tests, that would still be an improvement over where we seem to be. Test scores are falling. Why? It can't be that teachers are just teaching to the tests. Or, rather, if it is, they are doing a bad job at that, even.

I'm assuming the idea is that the kids are passing lower level tests better, but failing later tests? I'll take a quick dive on some of that later, to see if that hypothesis holds up.

> If the concern is that they are "teaching to the tests", my bigger concern is why are they so bad at that, even?

Again, I think purely teaching to the tests is short term gains for long term losses.

To take my example: I expect my students are going to be mostly 4's and 5's on the AP Microeconomics exam (this is good). But if I could have taught them why the functions looked the way they did, they'd have a better understanding of microeconomics. This would have been a dangerous thing to actually teach them, because if they used this knowledge on the test, there's no guarantee a pair of readers would understand it and give them the point. Better to dumb it down and teach the graph-based and algebra-based reasoning rather than the calculus-based reasoning of how it actually works, even though my students were capable of it.

I got to teach students stuff that they'll have to unlearn for deeper study, but maximized their short term performance. I'm totally willing to do this because it is in the students' interest: they really want a "5" on that exam to prove their capability to colleges and to skip a course. But if the test results weren't important to the students (like most of these standardized tests) but were important to -me-, that would be a terrible conflict of interest.

> I'm assuming the idea is that the kids are passing lower level tests better, but failing later tests?

4th grade scores kept climbing longer and have been flat for the past 7-8 years even as 7th grade scores have slowly slid downwards.

Right, but where are the short term gains? Scores are down.

The charts I saw on the NAEP site had 4th grade scores basically flat. Which chart do you see them going up in?

Specifically, their charts do show we are up in math compared to 90, but it has been flat since about 2003. And 4th grade reading has been flat basically forever, which is laughably in contrast with the older kids.

If your assertion is that they are having to teach to the test just to keep things flat, and that is why 4th graders are treading water and then starting to fall off by grade 8, I can buy that. I'm a little susp that the charts between 4th and 8th graders seem to not have a time lag on them. In that, whatever we are doing wrong seems to impact both grades at the same time.

If that is the assertion, though, I stand by my "if they are teaching to the tests, they are doing a bad job at that."

First off: I think this is a really silly set of weeds we've gotten into. Trying to eyeball an effect of an intervention from a time series is "what not to do in statistics 101." It's massive confounded: underlying student demographics have massively shifted

But if we're going to do it: If you increase the amount of effort spent "teaching to the tests" over the span of 2000-2010, and spot a rise in scores over that time and then a fall in older students' performance, then it sure looks like you've prioritized a short term gain.

> "if they are teaching to the tests, they are doing a bad job at that."

Well, test scores are massively higher than 1978, the first year that we have real data for. At the same time, it's not too obvious to me that we're really doing massively better than 1978. In which case: what are we measuring? What are we spending all this precious instructional time and effort on?

I'm smiling at that first. :D I really do think this is more "violent agreement" than it is otherwise.

I don't see an increase in 2000-2010, though? I see a jump in 2000ish, but the rest looks like noise. Even 2000ish is probably debatable, but if there is clear evidence that we made short term gains at long term loss, I'm game. My kneecapping here is that they didn't even get the short term gains. If you blame both on "teaching to the test" I will make no real defense of that. My attack is simply that whatever we are doing isn't working. In any real way.

That is, my main posit is that if they were at least making gains on the tests, that would be a gain we could see somewhere. But... we aren't seeing that. So much of the "teaching to the test" rhetoric is a moot topic without clearer evidence of short term gains.

So, yeah, violent agreement with your last paragraph. I have no idea what we are doing. Be it teaching to the test or otherwise, it isn't working. Evidence seems mostly flat for the past few decades. (Covid notwithstanding, as I feel that has its own causal factors that are beyond what we are talking about.)

> I don't see an increase in 2000-2010, though? I see a jump in 2000ish, but the rest looks like noise

There's a few points of improvement from '04 to '12.

The funny thing here is basically all of the drop we've seen since '12-- and especially post '20-- has been in the attainment of the bottom 10th percentile. Everyone else has been pretty flat.

> So much of the "teaching to the test" rhetoric is a moot topic without clearer evidence of short term gains.

We have gains on the tests during the mid 90's to 2010 timeframe. Big gains in mathematics. But does anyone seriously believe that mathematical literacy is much better?

Of course, it all depends what you decide is important. In the past 5 years that I've been following things, even very high performing students have fallen in arithmetical capabilities. (But maybe we just don't care if they're still doing higher math? I feel like I care)

I've seen the fall in arithmetical capabilities. But I've also seen that largely in myself. Oddly, I think credit cards and never having to make change for things is a big driver in that. To wit, you get worse at the things you don't have to regularly do.

Interesting point on the differences in the percentiles. I'd hazard that speaks more to exposure? Could be bias in what I was able to see, but I know every richer person I knew was able to get their kids in a private school, often with a personal tutor, during the '20 chaos.

I should say, thanks for sticking on this thread! Interesting conversation.

Reply on news.ycombinator.com