Hacker News Reader

Comment by yorwba | Hacker News Reader

Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but after analyzing the exam results they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again.

That means their experiment design is partially caused by their results instead of the other way around, which is a bad situation to be in. Their statistical analysis is completely inadequate for dealing with this.

And the change in engagement suggests that there's strong selection involved. Their attempt to use midterm scores to control for selection effects is unconvincing. Why not control for whether students used the platform more when there were only multiple-choice questions? Those are the ones who self-selected out of using the AI grader.

View on HN

I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid students to read books from the library just because you want to do research). Using engagement instead of results is a common choice of proxy to acquire more data, but it can become a vanity metric quickly.

> but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students.

This is definitionally impossible. Testing whether the tool benefits the students that get it is an attempt to benefit certain students at the expense of others. Get over this hangup if you want to do this sort of research.

Feel free to create your own university where you can do this sort of research in a discriminatory way. Have fun with the litigation afterwards. Also, explain to the peer reviewers that they just need to get over the hangup, too, when you try to publish your research.

"Feel free to create your own university where you have the funding for petri dishes. In the meantime, I'm going to publish without them, because I feel like this tissue culture I'm growing on my lunch is statistically significant."

Just because I am not personally entitled (by funding, by ethics, by career position) to collect the necessary data to demonstrate a hypothesis, does not make that hypothesis demonstrable with lesser data.

It's indeed difficult to do good education research, and instructors prefer to just mess around without a control group instead. But the intellectually humble thing to do would be to admit that the experiment can't differentiate between a method that turns bad students into good ones and one that merely identifies good ones through selection effects. When more conscientious researchers get around to doing a proper randomized controlled trial on an intent-to-treat basis, it invariably turns out to have been the latter.

> I agree with all the criticism...

Reply on news.ycombinator.com