Mechanical turks, AI now grading student essays | Hacker News Reader

Mechanical turks, AI now grading student essays

chronicle.com

35 pointsyummyfajitas14 comments

The real question - is it necessary, or even a good idea, to vertically integrate teaching and assessment? Maybe for a specific area of string theory, where both teachers and assessors are thin on the ground. But for calculus 101?

Exactly -- teaching and grading should be separated. I'd prefer a smart AI over some random remote assessor any day.

I always disliked the usual subjective way of grading, in particular if it takes several weeks to receive them. Having a smart program grade me instantly, and fairly, I think I would have liked it!

My wife is doing graduate work in Old English; most of her assignments to her students have a strongly quantitative nature: did you or did you not translate this correctly? Her peers, however, are in more murky territory. They study things like Gender and Being and Ideas of Progress in Post-Shakespeare England: strongly quantitative work. Hearing them talk at parties, the most common conversation is one of how they approach grading, being what rough heuristics they use. That some clever, overworked person is grading a essay means no more than having a heuristic markov process applied to establish some value for a given text but poorly owing to exhaustion and disinterest and arbitrary bias.

If we absolutely must peg a number to a person's work then we might as well have a computer do it reproducibly. I can't imagine there will be many University humanities departments happy to adopt such "inhuman" methods: best to wear actual humans down to nothing.

Please forgive the spelling issues and the last "quantitative" where I meant rather "qualitative". I've been struck by a miserable cold.

If the assumption is that homework and tests are designed to help the student learn the material, then subjective grading is very beneficial. For example, there are ways of solving math or comp sci problems (including simple "typos"), especially when you don't understand the rules properly, that will give a correct result, for the wrong reasons.

I would much rather a math or computer science teacher be looking through the process used to find the answer - and providing constructive feedback.

Just getting a quick yes/no response back seems a lot less helpful to me. I don't know how much the human graders mentioned in the article actually do in terms of feedback, but I imagine we're still a ways away from a program having insight into WHY you chose the wrong method or made the mistake you did. The program can't help tailor their teaching to help set you straight.

Of course, my view on this is likely clouded by the fact I went to a small school with small class sizes throughout. When you only have 5-20 people in each class, direct professor engagement happens. If I had to deal with TAs, I don't know how I would feel about the matter.

It mostly is separated anyway, with TAs doing the grading and professors doing the teaching. Of course, you could further outsource it from TAs to a 3rd party, but TAships have the dual role of funding grad students, so it might not save the university money to get rid of them, if they still plan to support grad students to the same extent as currently--- they'll have to find some other way to funnel that money to them. Of course, it might be part of a plan of reducing grad-student support.

> Software has no emotional biases, either

So what sorts of biases does it have? Remember the ol' adage about programmers figuring out any sort of metric, and exploiting it? I wonder if a group of students could find a way to game the grading system.

I know of some automatic grading system they tried for psychology students. It used some clever techniques to see how far a student's free-form answer was away from the model answer. The system gave essentially the same grades as manual grading.

But, the automated grading system did not take the order of words into account at all. If that was more than a test, it would have been easily gamed, or at least incited responses that having nothing to do with a human readable text. You would still have to learn the `passwords' for each question.

The essay-grading software I've seen described in enough detail to figure out how it works has a heavy bias towards enforcing specific structural rules, while mostly missing "quality of argument" type questions. So a student writing to exploit its bias would want to worry more about formal structure: essay organization, correct grammar, use of certain "must-have" keywords, etc.; but worry less about things like avoiding logical fallacies in their argument, because the software won't notice those.

It would be amusing if true AI rose out of our attempts to catch cheating students.

That would be fun, though I believe that the problem of catching students doesn't provide enough data.

This makes sense as it is a lot like how one is taught to answer essay questions on standardized tests.

> a "consumer-based approach" to higher education

That has to be another problem.

I actually think it'd be a good idea for education to become more learner-driven, but I agree it isn't a good idea without more changes. When you combine "consumer-based" education with an education system that is heavily oriented towards just-get-the-credentials, that's what makes it unworkable, because then what the customer really wants is just the credential ASAP. For learner-driven education to work, the learner (or "consumer" if we must) has to genuinely have things they want to learn, along the lines of what constructionist education, Montessori, unschooling, hacker culture, etc. try to aim for.

Reply on news.ycombinator.com