Hacker News Reader

Comment by smiths1999 | Hacker News Reader

smiths1999

I teach at the university level and this is what I have done with my exams. Everything is open book and in my experience there is no difference in the average exam score between open book online and closed (or open) book in person. It is also way easier for me to not have to worry about who is cheating and where everyone is looking (I also don't like the idea of forcing students to turn on their webcams).

View on HN

I finished up the coursework for my masters degree at the start of the pandemic. My university was quite flexible for how instructors would examine us, given how sudden everything had to change.

One of my courses, which only had about 8 students and two instructors, decided to do an oral examination, which ended up being basically a very in-depth, one on one conversation about the course material and based on the expectations set in the syllabus (so, no surprises).

While obviously not practical for large rosters, this was by far the best exam format that I have ever done in my many, many years of schooling. I'm sure not everyone would prefer it, but the students unanimously agreed to try it (wouldn't have done it that way otherwise), and it was just so great. It was not at all like an oral thesis defense, which was what I was a little worried about.

When I was an engineering student at the university of Naples, all my courses were examined both with a written exam and an oral one. No exception, no matter how many students. It was hard for us and the teachers but, boy, you had to really study that stuff! Since then I've become an academic myself and have been teaching in several countries. I have never found the same level of rigor in any place I've been.

I remember friends in the '80s studying law in Turin always having orals - as you say "they had to know their stuff"!

Italian universities get many things wrong, but in terms of quality of graduates I reckon they used to be really good. The downside was having a high number of drop-outs (of which I am one). They've gone through umpteen "reforms" in the last few decades so I don't know how they do these days.

> decided to do an oral examination,

This really is the best case, but as you note it was 8 students so quite manageable.

It requires a little skill on the part of the examiner, but you can quickly find out how much material the student knows with much higher accuracy than other exam formats, in my opinion.

One of the skills needed is to be able to make it conversational-feeling and reduce the anxiety of students. You can often tell when a student mostly knows what is going on but has misstated or misremembered something, and guide them around the place they got stuck.

Orals have a lot of advantages, but they also make it very easy for unconscious bias to come into play, in that all the criteria for grading are soft.

Good point, this is also one of the aspects of skill. There are techniques you can use effectively to mitigate this.

One unfortunate thing is that poorly done, orals can be very uneven.

There already is unconscious bias. You can see the student's name, their penmanship, their writing style, you likely know who they are, etc. An oral exam just changes things by changing the bias to accent, inflection, annunciation, skin tone, dress, etc.

When I was in school, 95% of grading was blind and nothing outside of exams was handwritten.

And while it was possible to de-anonymise, the academics were all in support of blind grading so why would they?

The only exceptions were projects where everyone was assigned a different topic, and graded presentations.

Of course, this is simpler in STEM subjects - it's not like you can guess someone's race or gender from their switching power supply design. Subjects that prize in-class participation and lengthy essays would probably be harder to blind-grade effectively.

This also depends quite a bit on class size. If you are one of 9 profs and 36 TA's on a whole year of a 1st year intro course, you can get together and batch mark finals very effectively blind.

If you are teaching a 4th yr/masters mixed class of 11 by yourself, you pretty much get to know who is who whether you want to or not. I suppose avoiding handwriting can help if it's appropriate (e.g., won't work on a math course) but I suspect you'll know everyones style by then anyway.

The examiner doesn't even have to use the oral exam to give a grade. They could use that part simply to figure out whether the student passes or fails. It's very difficult to cheat in an oral exam. Combine the oral exam with the written exam and you could get an overview of what the student knows.

This privileges confident speakers. (The same way written tests privilege confident readers, and standardized written tests privilege those who have the time and resources to study the standard.)

Schools should be teaching kids to be confident speakers and readers from an early age.

If teaching worked 100% we wouldn't need exams.

The real world also privileges confident speakers.

Not much if you do it well. Confident and wrong won’t get you far, either ...

I'm not sure about that.

It certainly could, but it also certainly couldn't. I imagine being confidently incorrect is likely to produce a worse result than being unconfidently(?) incorrect, for example.

Similarly, a less confident speaker may end up spending more effort justifying their answers, which could better expose their knowledge.

I think it would depend quite a bit on the examiner in this case. Some people may even be simply biased against particularly confident speakers, particularly considering the relative positions of the speaker and examiner.

It doesn't if the examiner understands the material. You can't bullshit someone who knows much more about a topic than you do - bullshitting with confidence will only make you sound like a fool.

If the examiner isn't much more knowledgeable about a topic than their students, then something else has gone wrong.

But you can appear to know less than you actually do through a lack of confidence.

Right, but that's why it takes a bit skill on the part of the examiner - you need to be able to support people through their nerves and lack of confidence.

> and it was just so great.

I'm delighted to hear that it went so well, and I am a believer in the idea. I have seen, from time to time, oral thesis defenses become rather tense and difficult, and think that things go better in proportion to the preparation of both student and examiner. Any general observations about what worked, for those contemplating giving exams in this fashion?

Probably the most important thing that made it a good process was that, although I was certainly under pressure to perform and show that I knew and could explain the material, the format was as a thoughtful, two-way, deeply-engaging conversation rather than a grilling, one-sided examination.

The examiner (one of my two instructors) had a list of questions/topics that we had to get through, but the specifics and flow were natural and spontaneous rather than artificial and forced.

What made the conversation good was that the examiner discussed points that I raised, raised points that I didn't, asked my opinion on e.g. real world implications of theories or conclusions that could be drawn if this or that were true, et cetera. This made it two-way and engaging. While he did not give me any answers, of course, when it was clear that I could bring up and sufficiently discuss a topic, then he would go into details, which would trigger even more detailed responses from me, and so forth. In this way I think he was able to probe the depths of my understanding while not needing to employ a one-sided question/answer format.

I think that it is difficult to bullshit a topic in depth with someone else who knows what they are talking about, so an oral exam probably does not need to be a hardcore opposition like a thesis defense might be.

The allotted time was maybe 30 or 45 minutes (I don't quite recall), after which the examiner would tell you what grade they would recommend. If it was a grade with distinction, then of course you would say thanks and be done. If it was less than distinction then you could request another 10 or 15 minutes of further examination to try to show your mastery. (I passed with distinction, so I didn't go through that part, but I assume it would have been a continuation? But maybe it would get a little more intense if you were trying to improve your performance at the last minute? I don't know.)

This is how exams are in Russian universities. You walk in. The table has a number of small paper cards on it face down with topics the course covered. You pick one at random, flip it over, and that is the question you need to answer. Since you do not know up front which you'll pick, you need to know all the material. Since you only need to answer one question, professor time is saved and exam throughput can be quite high.

Professors are also given quite a lot of flexibility in their grading. My mother had a fun story about a professor she had in college - a professor of a really hard math class who wanted to save on exam time. He announced "exam will be hard. Anyone willing to settle for a D, bring your report cards forward, I will mark them D and you can leave. No exam.". Some people came forward, got their Ds marked, and left. Once the door closed, he said "Anyone willing to accept a C, please come forward". Some did. After the door closed there, he announced to the remaining smiling students expecting easy As/Bs: "I'll see you all for the exam tomorrow 8 am".

No way this could happen in USA.

> My mother had a fun story about a professor

I've heard that story many decades ago in the form of a joke. It may have started from a professor who genuinely didn't care about failing students but did care about identifying the best.

My experience with the oral exams have been the opposite. The profs ask useless questions on details that do not matter at all and are never covered in a real exam. They are also very biased in how they assess you based on their preconceptions. And it doesn't give you the time and privacy needed to solve real technical questions. (Most of my written exams were only 4-5 questions in 3 hours, which is obviously not happening in an oral exam.)

In my alma mater for most courses we had to actually discuss what we wrote(and maybe some other topics) with the lector as part of the exam. Some were open book, some were closed. Regardless of that however when you get to talk to the professor it's very hard to fake knowledge and especially logical thinking. It sure takes more time(we usually had one on one for about 20-30 minutes per student), but you can size a person way better than just reviewing what he wrote/selected. And there is no obstacle(other than time constraints, which can be worked around) to continue doing this even during the pandemic. I think we are overestimating technology such as AI and its applications. Some things are just plain better when done by a person. I had a very positive experience with one professor who chose to round up my grade instead of down, because my errors were technical(carelessness mostly) not logical, so he then asked me questions from subsequent chapters to see my reasoning. I really didn't care that much about the grade, but that discussion and the overall experience of failures that could lead to positive change through introspective analysis is something that you can't get from an AI or any pipeline processing.

Thank you! I'm sure it takes more work to make an open book exam, but it's definitely to the benefit of your students.

If you can't see them, doesn't that open up the possibility of students taking the exam together?

I give regular online exams without any protoring software, pure honor system. I have three students who live together who consistently turn in identical work. What I don't understand is why the keep doing it, when I keep giving them 0s for copying.

Uncontrollable stupidity apparently. Where I went to school you would have been called to the Dean on the first event and perhaps out on academic probation if they didn't like your answers. Second time would have been academic suspension

Crazy stuff.

Presumably they have some kind of recourse as to their grades.

How did you find out they live together?

Not every university / university class is big. Most likely explanation is that they simply said they live together.

Ah, that'd make sense. I would just think that they wouldn't mention it if they planned on cheating lol.

Back when I worked first line at a University, I had access to the student records system (so that I could verify students if they required a password reset).

I was able to see current and past residence data, which I suppose could have helped me verify the student's identity ("what post code[0] do you currently live at?"), but I tried not to look at that information when possible.

Lecturers also had access to this system (it dealt with some submission data), so I guess they may have had access to view residency data?

This is just an educated guess based on my experience, though.

[0]: For American readers, think ZIP code

Reply on news.ycombinator.com