Universities should return to oral exams in the AI and ChatGPT era
theconversation.com
theconversation.com
Problem solved.
Expensive is your unsupported concern. The Greeks figured this out. Why assume this needs more expense than professor to do more than grade the performance that is carried out amongst what is some combination of peer and mentor?
Maybe a computer model is used to grade the performance. Maybe the mentor is a computer model.
the Greeks didn't have Psych 101 classes with 2000 students
People want to learn, but what they really want is a degree. No one is going to go two more years into debt for pedagogical purity
https://chem.wisc.edu/chemistry-building-project-overview/
> Chemistry, as the central science, is relevant to many other disciplines and is therefore part of the core curriculum of numerous degree programs on campus. By the time they graduate, 55 percent of undergraduates have taken a chemistry class at UW-Madison. As enrollment grew over the years, department and campus leaders noticed that students were often not able to schedule needed courses. This meant it took them longer to earn a degree.
There are about 10,000 under grads for each year at UW Madison. Chem 103 gets about 5000 students per year.
While that roughly matches the Psych 101 with 2000... this shows that "here's a real number."
Q1: Are there thousands of employers out there demanding students who have passed Psych 101 with flying colours?
Q2: If not, why not try to stop this?
I think alternatively we need employers to stop asking for degree’s altogether so the enrollees are people who are there to learn for themselves not for employers. It’d reduce the crazy demand and cost around education.
(Full disclosure: I don't understand the US college system at all [sorry!])
In what circumstances would a student be already at college and wanting/needing to "survey" psychology by doing Psych 101?
More broadly, who wants to major in psychology .. and why? Do employers need more professional psychologists?
2. “Survey” courses have 2 values through helping people learn new topics (1) gain broad understanding of the world, and (2) discover fields they may wish to major or study deeper in if they haven’t decided yet.
My caveat here is that I suspect (unsubstantiated) that some topics become filler by universities.
3. There are lots of jobs people do with a psychology major, besides a psychologist/researcher, but I don’t know how many of them require that deep knowledge explicitly.
Many jobs are probably better performed if you learn some psychology (eg a few classes worth) - marketing, non-therapeutic counseling (coaches, teaching, etc), HR, recruiting, doctors, nurses, organizational management, etc.
I want to reiterate point 1 - there’s value in education beyond employment, but it’s plausible that employers should care less. I doubt most employers expect candidates to take survey level courses, but requiring a degree of any sort implicitly assumes they have taken those courses.
OK, so what proportion of college students choose their classes for those reasons?
Thinking back to my time at college, there wasn't a whole lot of purely intellectual motivation (or genuine curiosity) going around.
The overwhelming majority were there a) to have a good time, and/or b) because they saw it as a route to a well-paying job once it was completed.
For example, oral exams are still very common in Poland and Polish universities. They typically takes less time to take, for a few reasons (the number of questions, the speed of writing versus oral production of answers, the ability of the professor to ask probing and follow-up questions instead of having to give the same battery of questions to each student in a blind and generic manner, even when it is clear that the student has answered the question in another answer or another way). The goal is for the professor to establish with confidence the degree to which the student understands the material.
Now, yes, it takes more time for the professor to administer the exam than it would take to administer a written exam, but note that grading exams is also very time consuming. You can delegate to grad students, but the same can be said of oral exams, so there's little differentiation there. The oral exam combines the administration of the exam with the grading, whereas written exams separate the two.
I have, yes. My university experience ran the gamut, in that I had both classes in a massive lecture hall of hundreds of students where most never spoke a word to the professor and all exams were done via scantron, as well as small graduate courses of around 4 students where the entire class was oral exams and one-on-one demonstrations and group discussions. The smallest course was only available at all because of a particularly passionate professor who insisted on holding it, and the class was never available during semesters where he wasn't available, because nobody else was willing to put in the work to personally administer the class. Notably, all those courses cost about the same amount of money per credit hour from my perspective, so the university was making orders of magnitude more money on the large class (where I can't see oral exams being feasible without significant investment) than they were making off the small class.
> You can delegate to grad students, but the same can be said of oral exams
It's much harder to delegate an oral exam than a written one. As you've mentioned, oral exams combine administration with grading, which means the grad students who are delegated to need to be able to administer the exam (ie, effectively probe the knowledge of each examinee) as well as evaluate their responses. In a written exam, the grad student only needs to evaluate the responses, usually against a set of predetermined criteria since the structure of the questions is set in stone.
None of what universities are moving towards is better, but I fail to see how it isn't cheaper.
But what if AI conducts it?
The point is that the written exam is largely outsourced to PhD students here. The professor only looks at the model solution before and the grade distribution afterwards in extreme cases. Also you can parallelize the grading so that even for 300 students the grading is done in a week. Typically the professor spends half a day otherwise it would be 2 month...
I only do oral exams for my course and I am very happy how well it works. I normally ask students to grade themselves afterwards and I am typically spot on (I have written down the grade before).
It's unfortunate because my mind becomes completely useless in those setting.
If knowledge can be easily acquired in 5 seconds, why are you testing people on whether or not they have memorized it? What good is 5 seconds? If the difference between an accredited engineer and a regular person is supposed to be a 5 second Google, then what the hell is the point?
We need to modernize assessment. We should have done it decades ago when cheating became rampant. Problem is that universities have zero incentive to fix this problem. Human resource teams don't either. The only people that really care are the people that have to work with everyone who gets hired despite having zero actual skills. Unfortunately, they have one of the quietest voices in this chain.
If you don't know much, you don't even know the right questions to ask your AI assistant, or the right tasks to have it perform. You'll also be less-able to check, and to apply, its output. The more you know, the more you can get out of your assistant.
At least so far, these tools are more like a bicycle than an internal combustion engine: they help almost anyone go faster and farther, but the people who are fit will still be able to go the fastest and farthest. Substitute "knowing stuff" for "fitness" and that's what we're looking at with "AI" (again, so far—if we achieve actual AGI so a huge set of complex tasks don't need a human involved at all, that may be another matter)
That's why people still need to know stuff they could find out in five seconds.
It sounds like you are agreeing with me here. We should not be making life-changing assessments of people's skills in unrealistic situations. In the real world, nobody gives you the question. Nobody gives you a breakdown of the tasks needed. And yet that is exactly how we assess people in these sorts of exams.
I agree with you on the AI part though. I like the bicycle analogy.
I cannot imagine a return to oral exams in the current academic setting with huge class sizes. The only place I could them being useful is in tiny honors/grad classes where the professor already has a close familiarity with each of the students.
imagine 15 hours of work being too much work.
Simply coordinating meeting with 30 individuals within a single week is really difficult--think about how long it takes a manager to deal with annual reviews for everybody.
The old German Diplom system (replaced by BSc/MSc about 15 years ago) was based on oral exams. The way this worked there was that there were much fewer exams. For example, in Computer Science I had my first oral exams two years into the degree, and there were only five of them: one in Mathematics, one in each of Theoretical, Practical and Applied, Computer Science, and one for the second subject. Each of these oral exam was 30 minutes but covered the content of four semesters. There was a second round of such exams at the end of the degree and that was it.
To be admitted to an oral exam, one needed to pass the courses that the exam was going to be about. Lecturer were free to set the passing requirements. It was often just something like a coursework threshold of 60%.
So he prefers spending that time grading written exams? Or does he just dump the burden on a platoon of grad students? (And if you're going to do that, why not have the grad students administer the oral exam?)
Oral exams combine administration and grading, so it can actually be less time consuming and less effort to give them. You can also vary questions based on whether previous answers already answer later questions, probing as you go along. The point of exams isn't supposed to be a bureaucratic checking off of boxes. It's supposed to be an assessment of familiarity with the taught material.
Oral exams offer a fluidity not possible in written forms and the rationale for a written exam -- whether by the stroke of fingers to key or hand to paper bound in Blue, it is now muted by the ability to video record the oral exam for later grading.
Thus the performance of the exam can be recorded; allowing for a leisurely assessment. With the performance itself capable of a far more robust introspection of one's unaided knowledge.
This might not actually hurt companies like blackboard or canvas that much, because universities are like these big cows where once they have an institutional license they will probably maintain it forever, even if they are also buying the competing product (typical to pay for microsoft business and google cloud services both even though they are entirely redundant for example).
This is exactly what the otherwise-fully-remote courses that I took did, about a decade ago. The school had arrangements all over the country (indeed, all over the world, IIRC) for testing locations, often at other universities (or at their own primary campus or a handful of satellite campuses), so as long as you lived within once-a-semester commute distance of one of those locations so you could do the final exam (I don't think they did a mid-semester exam at those, but it's been a while, maybe they did) you could take the courses, no problem. They had a proctor or two in the room to make sure no-one cheated and to help with basic issues that might come up, and the proctor collected your work at the end, to submit it.
People go through all of university, ace their exams, then go to get hired in their first job and faceplant the interview in spite of having the qualification.
This is no surprise, they haven't ever needed to practice that skill!
And how are you going to score students? Like autism with IQ160 that cannot talk but building Transformer model since elementary school something like that? Should you fail him/her just because doesn't like to talk and getting nervous in front of a bunch of examiners? And you sure you can score consistently against all students?
Of course human have been thinking about making a list of questions then ask a subset of it can somehow make the exam process more standardized. So you never think that you would end up asked the wrong question? And how it is standardized when you ask different people a difference question?
And guess what, those communicative sport guy or girls can always score higher than those introvert or even autistic self taught programmers. People just tend to judge others by the oral skills, not actual intellectual skills.
To be able to check that the student learned something.
Mostly it is the AI hallucination problem that stands in the way.
Furthermore, if AI continues to consume the world, who is interested in investing years into building up lifelong debt to only risk that everything you learned is now obsolete or irrelevant upon completion.
I agree, it's a great way to assess students. I've been through them too, they are nerve wracking, but end up being a lot better than written.
But the lawyers, man, they'd hate it.
There's too much variability. Some students would get 'harder' questions than others. Some would get 'grilled' more in depth. Some would be asked to turn around and face a whiteboard more. Some students would say this is because I'm part of a certain category. And of those students, a lot would unfortunately be in the right. Because professors are just as biased as the rest of us, if not more so.
So, if you'd want to do this, the lawyers would say you'd need to have at least one person of every imaginable category in the room giving questions. And it would need to be an open exam (Pence rule), and they'd like for the exam to be recorded too, video and audio. And you've got to use a rubric with standard questions, so we can make sure there's no bias. Which means you have to sequester those that completed from those that haven't, phones included. And the proctors have to be behind a screen. And a few dozen waivers, just to round things off.
As others here have said, the expense would be huge. At least a handful of people, A/V techs, a revolving door of students, translators, the works. All just to end up asking students the exact same questions anyway.
I agree that it's more subjective, but with verbal communication (where you can have back and forth), it's easier to AVOID miscommunication.
Both oral and written have their pros/cons. Perhaps the best approach is to rely on written for scoring and oral for "sanity checks" (i.e. is the person just faking it).
You're not supposed to "miscommunicate" by the time you graduate.
College students in the 1910s could read Greek and Latin. Before beginning college. And there were no fancy apps to help them. Everything was by the book and hard memorization. But it would be asking too much of a graduate in 2023 to be fluent in English?
A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.
I think in the era of remote work one might assumes that all communication is done via slack/email and no real time verbal communication is needed.
The core of the problem lies with your expectation that just being fluent in English is sufficient to not have miscommunication. Communication is a lot more involved and something as trivial as not having a good night’s sleep can take your attention away from what others are saying. Written communication helps because you can read something back quickly if you skipped over something or were not paying attention.
Specialization is the only way we can currently make progress as a species, and is the only way we will be able to for the foreseeable future.
If everyone only had a passing or surface level knowledge of every field, they could not add to our collective understanding of any field.
To actually advance human understanding in a field, an individual needs to catch up on everything that has been learned about it up to this point. For some fields, that is hundreds of years of research and knowledge gained. That's not practical to learn for more than one or two fields - we simply don't live long enough.
If you want the person who makes the next alloy that makes planes another 20% lighter, you're going to need someone who has dedicated most of their life to just learning material sciences and is putting all their effort into that topic. It isn't going to be done by someone who had one quarter of a semester in college on it. (With rare exceptions)
Me thinks you are mistaken.
In case of verbal miscommunication, you can always give the other party a second chance and ask them to repeat or rephrase their statement or you can rephrase yourself if you felt your originals statement was confusing.
In case of written communication there is no second chance to rephrase your statement. What's written is written, and that goes, regardless if it was well understood or not.
AI won't get them right, but sadly probably more than half of the humans won't as well.
Another thing is that huge part of the knowledge passed in schools can't serve as a basis for any sensible inference.
This could hurt remote learning, but if you are remote there are plenty of ways one can cheat on their exams.
If anything their advantages over oral exams become higher.
I'd have to ask for a typewriter.
The student has to prove to the machine their capacity for intelligence on subject mater.