So what sorts of biases does it have? Remember the ol' adage about programmers figuring out any sort of metric, and exploiting it? I wonder if a group of students could find a way to game the grading system.
But, the automated grading system did not take the order of words into account at all. If that was more than a test, it would have been easily gamed, or at least incited responses that having nothing to do with a human readable text. You would still have to learn the `passwords' for each question.