Reading Incomprehension

At the center where I graded, only 1 in 14 essays were read by a second scorer for verification. "Calibration" consisted of a half-hour of reading pre-scored essays and taking a test that required a passing rate (being within some margin of the "true" score) of only 80%.

There were ridiculous formulas about the number of adjectives, etc., required for a certain score. Imaginative writing suffered the most; cookie-cutter writing was rewarded the most.

Scorers worked eight hour shifts. Sometime after my morning coffee I was the most generous, and in my afternoon lull, when I could hardly keep my eyes open, I'm pretty sure I graded the hardest.

I remember one of my colleagues had, sadly, suffered brain injury in a car wreck. His scores were all over the place, as I could tell when he asked my opinion on the essays he was scoring. Others had modest educational attainments -- people who had maybe gotten a bachelor's degree twenty years before, but not really been engaged in mentally rigorous work in the mean time. Many of them couldn't reliably spot an adjective. They were amazed that I had an "accuracy rate" of 97%, the highest at my table by a good 15 percentage points.

We were grading fourth-graders' tests from Arizona. The previous cohort at my center (that I didn't belong to) had graded high school tests from, if memory serves, Minnesota. (This was late '90s). The tests were "high stakes" -- failure on the test kept the student from high school graduation. It later came to light that some large number of students who had failed the test and been denied graduation, had later challenged their scores and successfully shown that their tests had been severely mis-graded. They were allowed to graduate, half a year too late.

After the grading period ended, I stayed on a mailing list to hear about job openings the next summer in that city. However, some time during the spring I got a letter saying that there wouldn't be a grading center in our city that year. Wonder why.

Last year, my wife and I toured the local Montessori school, and I asked whether students would be required to take the Ohio state-wide standardized tests. The teacher, thinking that I was concerned that their "performance" would not be measured, said that they could optionally take them. When I explained my relief that their time would not be wasted, she chimed in about how much she hated the tests as well.

American society is so obsessed with measurement that it ignores the nuances of what it seeks to optimize upon. By definition, any outstanding achievement lacks a convenient standard of measurement.

Of course this would increase the cost of grading such tests...