The Algorithm Didn’t Like My Essay
nytimes.com
nytimes.com
The increasing reliance on automated tools and standardized tests is just one of many symptoms of the increasing detachment between teachers and students. We have to stop doubling down on the prussian style educational system, it stopped returning dividends decades ago. More so we have to stop acting as though we can just solve our educational problems by throwing more money at it. We need to empower good teachers and parents, we need to encourage and support students. Right now all of the incentives in the system run counter to those things.
I've heard this (anecodotal) argument alot. Is there any empirical backing for it, and within what parameters does it hold true? My mother taught at a very expensive private school, which enabled me to switch from a lousy public school . The difference in the quality of teaching was dramatic. Private school teachers don't get paid alot more than public school teachers, and often less, although they are compensated with fringe benefits like I enjoyed, and prestige. In any case, elite private schools certainly get better results, which is why rich people pay alot of money to send their children to elite private schools. (Lots of this is attributable to the selection process, which is part of what parents are paying for.)
In contrast, public inner city schools can't choose their student population, and lack basic things like, books, paper, and air conditioning in the middle of the summer. I'm sure that proper and equitable funding of public education wouldn't solve all the problems in US public schools, but starving them of much needed resources certainly exacerbates the problems they do have.
I personally think that explanation fits both the "cost is orthogonal to education quality" argument of the parent, and your point that elite private schools have significantly better education quality, though I do not know offhand of studies about this.
I'd guess that the same or less is being spent on actual education today and the excesses, whatever they may be, are simply siphoned off by the parasites in the system.
It doesn't seem to matter how much $ per student gets added, teachers still end up buying classroom supplies with their own money. The system just has more $180k "administrators" now than it used to.
It is clear that it is going on an internal dynamic until those dynamics finally conflict with reality. That collision will be..painful. Given the crushing levels of student debt in this country it may happen sooner rather than later.
If I were in charge, I'd institute a simple change. Improve financial aid. Beef up the federal loan guarantees. Only make either available at schools whose tuition was in the bottom 70% when you applied. With that incentive, I think that a lot of schools would find ways to reduce tuitions towards what (inflation adjusted) was more than sufficient several decades ago.
Accounting for inflation and number of students, university budgets have not been going up 10%/year; at some universities, the real per-student cost of higher education has actually been declining. What's been going up at 10%/year is the tuition "sticker price" for students who receive no scholarships or need-based aid. But this is offset by two contrary trends: 1) a decrease in the percentage of students who are actually paying the sticker price, especially at private universities; and 2) large decreases in state funding for public universities.
Consider the University of California system (all figures below in 2012 dollars). In 1990, it spent $21,000 per student (dividing its total budget by its total enrollment). Today it spends $16,500 per student--- a decrease in the cost of education of about 25%. Tuition has nonetheless gone up, because the state-funded portion of its budget has declined even faster: from $16,000 per student in 1990 to $9,500 per student today, and possibly to $8,500 per student in the coming year. As a result, the student-funded portion has risen from $5,000 to $7,000 and soon $8,000 per student on average (with a sticker price a bit over double that).
If you want the UC system to be able to run on tuitions that were sufficient in 1990, you'd have to also return state funding to what it was in 1990...
Writing is an artform and as such should be treated and graded as one. Sometimes you need to use language that is grammatically incorrect to get a point across and software is not nuanced enough to pick up on this. I can see this helping to turn out students that are good grammaticians but terrible writers - optimizing for the wrong outcome.
In CS, we often grade assignments automatically, although we expect students to write the assignments by hand. Although there are many classes where this would be a bad idea, there are some where it works quite well - if the problems are strictly specified, grading an assignment with what are effectively unit tests is reasonable.
Grading and doing an assignment are different classes of problems - roughly mapping to decision problems and search problems. Just because the grading is computationally tractable or acceptable does not necessarily imply that the act of writing the assignment is.
I agree with your second paragraph by and large. There is a time and place for automatic grading, and most writing should not fall into that category.
It's probably possible to grade student programs like this essay-grading system works: instead of feeding the programs to a compiler and running against test-cases, just turn the source code into a bag-of-words representation and feed it to a statistical classifier. There are almost certainly some statistical correlates between the source code of programs and whether they're good/bad submissions. But the results would be both less reliable and less informative as feedback.
Edit: basically, the sheer scale of the grading required for a standardized test means that it's never going to be a real test of writing skills. It wouldn't surprise me if overly skilled writing is actually a negative even with the human grader—I could easily see a rushed grader misreading something clever and giving it a low score.
(The wisdom or effectiveness of these tests in the first place is a whole different discussion.)
I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. Add in a factor for essay length, unique word cound, and grammar errors, and you'd probably get pretty damn close to perfect, within a reasonable statistical tolerance.
Children don't produce art, almost all of them produce garbage, right up to the top 1%. Make no mistake, all that standardized tests aim to measure is how stinky the shit that they produce is. I'm fine with missing a bit of brilliance at the top; those students will do just fine anyways. It's ranking the relative crappitude of the middle that's important here, and I think the standard (if crappy) grading procedures do a pretty good job here.
The best part about that is the fact that people misspell words even with spellcheck! Come on, how hard is it to notice the squiggly lines and correct your error by picking the correct spelling from the drop down list?
And I wouldn't say what I produce is shit, but then I would have said that when I was twelve. And I mostly have to double time on projects and essays, so they probably are shit.
EDIT: And since losethos (Schizophrenic dude who's hellbanned.) asked, the boss; it's pretty clever to use someones tendency for pedantry against them.
I say this as someone who graduated at the top of my high school class - if you can't get me to work hard for you, you're pretty screwed with the average student. It's no wonder that the stuff they wrote for you was crap.
What he said: "Most of these kids are straight-up idiots..."
He hasn't already decided, he just came up with a prior.
I would recommend downloading the dataset[1] and taking a look at some of the essays. There is a tendency for an educated audience to (justifiably) worry about exactly what you've articulated, but when you read through the essays, you get an appreciation of how valuable a tool like this could be. These are elementary and middle school essays rife with rudimentary mistakes (which are at times extremely charming) and generally not filled with subtle arguments prone to misinterpretation.
If algorithms can help every seventh grader in the country write a solid 5 paragraph essay, that's progress.
I wrote about this more extensively here: http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-ma...
Two things: 1) Agree about the explanation bit. In fairness, that was not part of the competition. But I do believe that, given the way the features were constructed, many of the algorithms could be modified to also provide "explanation" of their scoring. The value is highly dependent on the features though (length, for instance, would probably be a prominent "explanation"), and I haven't actually tried it.
2) The "This would grade Dickinson/Hemingway/DFW/etc poorly" argument is true but, again, read the essays. These are serious edge cases - ones that should be addressed (by, for instance, correlating algorithmic scoring with human scoring), but this argument doesn't diminish the applicability of the algorithm to the vast majority of essays and students.
If a student used an essay-writing algorithm he designed himself, I would certainly consider that okay.
Company B will correct their software, but meanwhile Company C has come along and introduced a sophisticated analyzer which is able to grade the presence and quality of the logical argument being made in the essay. Company C will demonstrate how Company A and B's software grades as "Excellent" an essay which is correct English and isn't gibberish but makes no logical sense.
After an indeterminate number of iterations with the software getting better and better at finding issues, the only way to game it will be to write a great essay.
At least for this training set, my algorithm rewarded the length of the essay most of all (something like 65% of the total prediction). The only other significant factors were misspellings and prevalence of certain parts of speech.
That model matched the accuracy of human graders and several commercial essay grading packages.
Students reverse-engineering comparable algorithms won't necessarily have to write well to score well.
Or, if it's slightly cleverer than that, certainly you could defeat it by producing a single, perfect-length, stylistically fantastic essay... which would be regurgitated word-for-word regardless of the subject matter.
I think, mind you, that software does have a place in analyzing student essays. If I could scan in an essay and have it spit out a word count, highlight any spelling errors or potentially problematic turns of phrase, and [most importantly] analyze for plagiarism, that would be valuable.
in contrast, i suspect algorithmic approaches can be gamed more easily because they don't adapt in the same way. they're not solving the hard ai problem; they're grading essays (currently) written for a human reviewer.
for example, what happens if a child learns an existing text by heart and then substitutes appropriate nouns and verbs to suit the context? say they learn "We hold these truths to be self-evident, that all men are created equal" and then, for an essay on their favourite pet, they hand in "We hold these kittens to be furry, that all kittens are created hungry". That's good grammar; it's got suitable references to the subject; it's clearly nonsense.
For 99% of the high school students I've worked with, their essays would be better if they had to pass Microsoft's spelling and grammar checks before they were submitted. The exceptions were all exceptional enough that they knew that they no longer needed to listen to those automated filters.
If the grading curve of an essay is a checklist how is this finding a surprise?
They'd found it was one of the easiest text classification problems that had ever been tried. With a very small training set they'd attained off-the-charts accuracys.
When reading articles on the Internet I look out for superficial factors including:
‣ Misuse of U+0022 typewriter double quotation marks in place of U+201C and U+201D double quotation marks
‣ Misuse of U+002D hyphens in place of U+2013 en dashes and U+2014 em dashes
‣ Poor understanding of comma and semicolon usage
‣ Choice of typeface, use of ligatures and micro-typographical features, amount of leading used…
‣ Misuse of their, there, they’re, where, were, we’re, affect, effect…
‣ RAS syndrome (sīc)
These factors can assist with analysis of writing. An author can generate credibility in certain contexts by using triangular Unicode bullets instead of ASCII asterisks. Use of lengthy and complex words in place of simple English can also be mistaken for credibility. Analysis of these techniques will show that the writer is often attempting to belittle readers. Documentaries utilise similar tactics that the untrained eye will likely mistake for credibility. Soft filter effects, use of bookcase backdrops for expert interviews and silence have profound manipulative effects on viewers.
Grading algorithms will likely reward use of manipulative and belittling writing techniques and penalise honest superficial mistakes. I would rather read the honest opinion from an author that misuses their and there than have to carefully analyse the carefully crafted writings of a dishonest linguist.
As an aside, I recommend bookmarking The Browser[1] (Writing Worth Reading). It would be interesting to see the results of algorithmic essay grading against this curated collection of articles from across the Internet.
I'm far from well informed, but my understanding of standardised tests is that the standard specifies the algorithm, which already ignores your good points above to achieve standardised grading. All that really changes is whether a human or a robot executes the algorithm, the human insight has already been squeezed out of the system.
There are a couple of things to realize about this challenge:
I wouldn't conceptualise the challenge as trying to find features of good essays. It is more about trying to find features that are predictive of the essay being good.
This is a subtle but important distinction. One example is that the length of essay was hugely predictive of the score the essay would get - longer meant better.
Is a longer essay really a better one? No. But, at the level the students were at, it just so happened that the students who were able to write better, also were able to write longer essays.
While you get good accuracy using techniques like this, its debatable how useful or robust this general approach is - because you aren't really measuring the quality of the essay, so much as you are trying to find features that just happen to be predictive of the quality. Certainly, it would seem fairly easy for future students to game, if such a system was deployed.
This isn't a general attack on machine learning competitions - but I wonder, if for situations that are in some sense adversarial like this (in that future students would have an incentive to game the system), whether some sort of iterated challenge would be better? After a couple of rounds of grading and attempting to game the grading, we'd probably have a more accurate assessment of how a system would work in practice.
There is another important feature of this essay grading challenge, that should be taken into consideration. There were 8 sets of essays, each on a different topic. So, for example, essay set 4 might have had the topic 'Write an essay on what you feel about technology in schools'. To improve accuracy, competitors could (and I would guess most of the better teams did) build separate models for each individual essay-set/topic. This then increased the accuracy of, say, a bag of words approach - if an essay-set 1 essay mentioned the word 'Internet' then maybe that was predictive of a good essay-set-1 essay, even though the inclusion of 'Internet' would not be predictive of essay quality, across all student essays.
Its important to remember this when thinking about the success of the algorithms. The essay grading algorithms were not necessarily general purpose, and could be fitted to each individual essay topic.
Which is fine, as long as we realize it. The fact that it was so easy to surpass inter annotator agreement (how predictive one of the human graders scoring was of the other human graders scoring) was interesting. Its just important to realize the limits of the machine learning contest setup.
I would guess that accuracy would down on essays of older, more advanced students, or in an adversarial situation where there was an incentive to game the system.
I'm not able to dig up the name, but there's a named effect in statistics (especially social-science statistics) describing exactly that. When you find a correlate of a desired outcome that has predictive value, a common result if you then set the correlate as a metric is that a substantial part of the correlation and predictive value quickly disappears, because you've now given people incentives to effectively arbitrage the proxy measure. You've said, I'm going to treat easy-to-measure property A as a proxy for what-I-really-want property B. Now there is a market incentive to find the cheapest possible way to maximize property A, which often ends up being via loopholes that do not maximize property B. A heuristic explanation is that proxies that are easier to measure than the "real" thing are also easier to optimize than the real thing. At the very least, your original statistics aren't valid anymore, because you measured in the context where people were not explicitly trying to optimize for A, but now they are doing so, so you need to re-measure to check if this changed the data.
It could also be the case that length is one of the features your human graders are using to grade essays. I.e., it might really be causal, rather than merely correlated.
In my (anecdotal) experience, teachers certainly do this. While in college I developed the skill of utilizing excessively long and verbose language while elucidating simple points simply to incrementally increase the length of essays [1].
Luckily a great prof in grad school (thanks Joel) beat this bad habit out of me.
[1] In college I learned to pad my essays with verbose language.
However, the kaggle supplied 'straw-man' benchmark, which worked solely based on the count of characters and words in the essay, had an score of .647 with the training data. (The score metric used isnt trivial to interpret - it was 'Weighted Mean Quadratic Weighted Kappa' - but for reference the best entries had a score of ~.8 at the end)
The score of .647, just using length, is quite high. For length to have this powerful a causal predictive effect, the human graders would have to be weighting for length, as a feature, very heavily.
I can't rule that out; but I think its highly likely a major component of the predictive effect of length was correlative, rather than causal.
Well, I'm hardly surprised that an algorithm can duplicate that kind of superficial marking of an essay!
Good heavens, what are you doing to your children in the US?
To prove this point, I suggest setting up a testing competition, this time for algorithms to generate meaningless 'essays'. I expect their easy success obtaining full marks.
So, what will be the educational outcome of this? In three simple words: more dumbing down.
Is this another example of the obsession of officials everywhere with the pedestrian form at the expense of original content? I think it tells us a lot about their own limited mentality.