Comment by doctorpangloss | Hacker News Reader

Comment by doctorpangloss | Hacker News Reader

Parent Full thread

i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point.

let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test.

overall, it would be surprising if AI usage impacted your Wechsler scores. someone has done this study and the impact is quite quite small. BUT. do we care? We don't use Wechsler scores for admissions, we don't use them for jobs, we don't use them for... are you getting it now? A Wechsler family test is measuring something real, just like a university exam measures something. But what do we USE them for? Wechsler and a typical university exam are, in some senses, EQUALLY vague in terms of their fitness for purpose for answering a question like, "should we hire this guy?"

Like there is an association between IQ and earnings but it is actually surprisingly small! There is an association with math education and earnings and it is also surprisingly small. And consider how many people get by just fine without using a single piece of math education once they have finished school - like what if maximizing your earnings isn't all that it is about? Are you getting it now?

The issue isn't the AI usage. I can find tests that are immune to AI usage. The issue is using tests for things that they are not designed for. We pick and choose, for some subtle but nonetheless pervasive cultural reasons, which tests we use for which purpose, and very frequently, not because they are calibrated for the chosen purpose. This is coming from someone who scores very well on all these tests, and have kids, so I have a very strong incentive to buy into the status quo, and I'm telling you: academic testing has been fucked up for a long, long time.

Education is a complex topic. I don't claim to understand it, nor do I think we can sort this out in a hacker news thread. What I reject is the simplistic claims like "exams don't mean anything". There are certainly exams that are badly designed, but there are also exams that are well designed. A well designed exam can look very badly to different people, based on what they know and where they come from. It's a bad idea to think exam scores as the single metric of education quality, it is equally bad to reject them completely, because chances are any alternative measurement people come up with are going to be worse.

I think this deviates from the original topic. The article's intent is to argue that AI helps with learning itself, while you question whether this sample can serve as a reference. But what if the target of this sample is a group of serious college students? The article questions AI's enhancement of learning ability, not whether that exam has a real purpose.

You're right that academic exams also measure something along the lines of instruction following / obedience / willingness to jump over hoops for no good reason etc. and that's often a good signal for most kinds of jobs.

I just started teaching undergrad CS courses after ~20 years of various non-academic jobs and it took me about two semesters to realize that almost nothing matters except how I assess students.

Two weeks of ADHD-fueled research later, I concluded that academia is actively resistant to implementing assessment reform because it would expose the utter pointlessness of most of what happens in university classrooms.

The reality is that we have no idea what most university exams measure because they are ad hoc, written by amateurs (yes, most professors are untrained in pedagogical methods) with zero psychometric validity analysis.

> almost nothing matters except how I assess students

This is a reasonable opinion after two semesters.

IMO, with more experience you should have modified your stance.

If all classes were pointless, students would leave at the same point they entered. Since we observe that is not the case, something is happening in class. Do more of that.

You’re conflating the course and the classroom.

The explicitly stated goal of most courses is mastery of a well-defined set of domain-specific knowledge and skills. The only way to objectively determine if the course achieved that goal for a given student is comprehensive assessment. Bad assessment design, much like bad experiment design in science, is worse than useless—it actively misleads and confuses.

A well-designed, repeatable, reliable assessment is a prerequisite before one can even consider the effects of particular pedagogical methods inside or outside of the classroom itself.

Reply on news.ycombinator.com