Hacker News Reader

Comment by andreyf | Hacker News Reader

I wonder what limitations there are to using standardized testing for evaluating "progress"? From personal experience, many of my friends who score very well on standardized tests aren't necessarily "smart" in the sense Feynman was "smart" - that is, capable of creating and sharpening their models of the world.

What's ironic is that while Teach for America acknowledges that it's extremely difficult to measure what makes a "good teacher", in that same sense of a person who continuously creates and sharpens their models of the world, they seem to assume it's trivial to measure a "good student", even though the two are fundamentally the same.

View on HN

I wonder what limitations there are to using standardized testing for evaluating "progress"?

Any heuristic you care to use is significantly better than the traditional metric for school quality, which is spending per pupil. (This metric was selected because optimizing for it optimizes for the outcomes teachers unions desire. American public school exist to employ union teachers and sometimes cause education as an industrial biproduct.)

This puts me in mind of the metric my day job used to track productivity for several decades: hours worked by the engineering staff. That one had perverse consequences, too. (Having a bad quarter? Need a good statistic to report to the head office? Start a deathmarch!)

Goodheart's Law (not to be confused with Godwin or Murphy's Laws): "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes".

Randall Munroe (xkcd) has make a similar observation.

As soon as you start measuring something and attaching organizational outcomes to the results, you encourage gaming.

As above, better that what people are pursuing is student learning than spending per school.

It's not a teachers job to teach "smart". It's a teachers job to teach the subject, which gives the students more tools to make themselves "smart". I'm not sure if there is any way you could design a schooling system that teaches "smart", short of outlawing homework so students have more time to teach it to themselves. (I'm only half joking).

There are a number of levels in teaching (and learning). You learn the basic facts (or processes), you learn to apply them in simple situations, and you learn to apply them creatively. This can be tested by multiple-choice tests, or more in-depth tests, but multiple-choice is fairly reliable and very cheap.

Besides, the difference between a "good" and "bad" student is pretty easy to measure, if you look at the broad population.

It's not a teachers job to teach "smart".

Yes it is, by definition. Once people are "smart", in the Feynman sense, there's no need for anyone to teach them a subject - they're self-motivated learners that will seek them out on their own.

You learn the basic facts (or processes), you learn to apply them in simple situations, and you learn to apply them creatively.

Although somehow intuitive, I don't think that's a good model. What are "basic facts"? What is a "simple situation"? What does it mean to "apply facts creatively"? All of these are vaguely intuitive but not well defined, with no epistemological basis. There is a much better definition of learning - the ability to create models of our experiences, test, and refine them. I know of no reason a cheap test can't be devised to test such learning, nor of any attempts to create one.

We seem to be at loggerheads on two points.

1 - I'm focusing on basic proficiency (a bottom up approach - you learn the basics then the hard stuff is easy), you are focusing on learning through break-through discoveries (the top-down approach). (I may be creating a false hierarchy, but I'm going to run with it).

2 - I am focusing on teachers directly teaching students, while you focus on teaching students to be self-motivated learners.

Now, I can't think of any way to teach self-motivated break-through learning. The closest thing I can think of is to make subjects interesting (getting teachers who are enthusiastic about the subject, and providing interesting text books (yeah-right)), and give the students free time to pursue their own interests (which is why I half-jokingly suggested outlawing homework). The other model I've seen (don't teach properly so students have to learn) seems a little perverse to me.

(oh, the basic facts -> simple application -> creative use? Fact - past tense. Simple application - what did Spot do? Creative (or complex) use - Why did Spot do it?)

Look up Montessori.

You're right about #1: reading, writing, and counting. But I think it stops there.

Addition and multiplication can easily be self-discovered, and, if they are, expanded to exponentiation, tetration, and Knuth's up-arrow notation [1]. The recursive relationship between x+...+x, x...x, x^...^x, etc isn't beyond most kid's minds.

Regarding #2, again, let's not be vague. Let's define teaching as creating motivation to learn - through grades, or social rewards, or whatever. Let's define "learning" as the acquisition of models. Using this model, the role of a "great teacher" becomes obvious - it isn't transplanting a models from one mind to another, but motivating the creation and evaluation of one's own models.

I'm glad you brought up "past tense" as an example of a "fact". I'll agree that we have a common experience of time (and hence, "the past"), but will note that communicating about past events is innate (no need to learn it). However modeling verbs as "past tense" or "future tense" is a trivial linguistic model - a intuitively comfortable model, but without much evidence - incapable of describing reality - in this case, our innate abilities for language.

1. http://en.wikipedia.org/wiki/Knuths_up-arrow_notation

The article definitely glossed over how they measure "good students" and this is a very real problem. When student success is measured only or mainly by standardized tests (and teacher and school success extrapolated from that) surprise surprise: teachers start teaching only to the test. Now you have students that are great at taking tests but have not been encouraged to develop those "Feynman smarts."

Now you have students that are great at taking tests but have not been encouraged to develop those "Feynman smarts."

Proof that one goal necessarily excludes the other? Right now we are getting neither, in most cases in most schools, so what Teach for America is looking at seems to be progress to me.

Sure these things don't, by definition, implicitly exclude each other, but in practice they do. How much time do you think teacher's have in a day? For proof talk to any good teacher, or sit in on a school board meeting.

Reply on news.ycombinator.com