Hacker News Reader

Comment by simonw | Hacker News Reader

The first comment on that article caught my attention:

> I am a grad student in the philosophy department at SFSU and teach critical thinking. This semester I pivoted my entire class toward a course design that feels more like running an obstacle course with AI than engaging with Plato. And my students are into it.

It would be interesting to take a class of students and set them an assignment to come up with assignments for their fellow students that could not be completed using ChatGPT.

About ten years ago I went to a BarCamp conference where one of the events was a team quiz where the questions were designed to be difficult to solve using Google - questions like "What island is this?" where all you got was the outline of the island. It was really fun. Designing "ChatGPT-proof assignments" feels to me like a similar level of intellectual challenge.

View on HN

> Designing "ChatGPT-proof assignments" feels to me like a similar level of intellectual challenge.

Designing a "mostly ChatGPT-proof class" is in my opinion actually rather simple: just be inspired by the German university system:

Each week you have to complete exercise sheets which are often hard. But being capable of solving a certain quota of the exercises is just the beginning: this only qualifies you for being allowed to take the actual (oral or written) exam.

In this sense (as many professors will tell you), the basically sole purpose of the quota for solving (often hard) exercise sheets is actually preventing students from the "self-inflicted harm" of doing an exam that they are not yet prepared for.

And yes: while it is forbidden to "cheat" (e.g. ChatGPT) on your exercise sheets, this policy is typically not strongly enforced (you will nearly always just get a unequivocal, strongly worded oral reprimand by the tutor). Instead, if you did this, you will for sure not be prepared for the upcoming exam, and flunk it (and most students are very aware of this).

Oh yeah, I didn't mention yet that if you flunked the same exam typically 3 times (depending on the university), you have "finally failed" (endgültig nicht bestanden), and are not allowed anymore to study the same degree course at every German university.

> I didn't mention yet that if you flunked the same exam typically 3 times (depending on the university), you have "finally failed" (endgültig nicht bestanden), and are not allowed anymore to study the same degree course at every German university.

When Europeans wonder why they are falling behind the United States both economically and technologically (Especially in the AI race), here's a perfect example of why. A culture that turns you into a some kind of a permanent failure because you failed to check a box some arbitrary number of times isn't one that produces innovation.

Of course, the next European neighbor country where you can continue your studies (even taking most of your credits with you) is often only a few hundred kilometres away.

If you really think that a university degree course is still the right thing for you, you typically just choose another "related, but different" degree course at some university in the same country.

Concerning "next European country", on the other hand, keep in mind that in this country often a different language is spoken; in particular for your daily life, you cannot assume that everybody understands English or even your native language.

And when Muricans wonder why the rest of the world laughs at your simplestic greed based view of "winning", this comment is a perfect example of why.

What a weird way to try and connect inter continental economics and private company valuation as a sole metric for success or achievement with test taking at a university level in Germany...

It's weird to correlate how the idiosyncracies of a deeply-rooted developmental system might affect the sensibilities or perspectives of the adults that said systems develop?

I want to agree with you. In fact I do agree with the point you're making about how policies, cultural values, and outcomes are intertwined.

However you need to realize that the country you're holding up as a counterexample brands people as felons over some fairly absurd things and more generally has a remarkably dysfunctional justice system that exhibits precisely the same cultural failures that you're objecting to here.

> Designing a "mostly ChatGPT-proof class" is in my opinion actually rather simple: just be inspired by the German university system:

Now the real question is the costs. In that, can you somehow make this system work at scale with the same profit as before? I know that the German system is a lot different than in the US, including things like tracking in 9th grade and below. But the real question is if the Universities make as much, if not more, cash. Because if the answer is 'no', then it's unlikely to be adopted.

> Oh yeah, I didn't mention yet that if you flunked the same exam typically 3 times (depending on the university), you have "finally failed" (endgültig nicht bestanden), and are not allowed anymore to study the same degree course at every German university.

That is an insanely awesome and clever idea. I love it. It puts real stakes there, if only perceived ones. I imagine that in the US if you flunk out of a major's classes twice or more, the number of students that continue on in that major is probably pretty low already though.

It's not clever, it's pointless red tape. I don't know if it's different in Germany but at least at US universities most higher level courses are part of a sequence and are thus only offered at a specific time of year. Thus failing delays your degree by a year and creates all sorts of logistical issues for the student.

Doing that more than 3 times would be absurd, and anyway the sort of student who has 3 F's on his transcript within his chosen major is unlikely to be maintaining a GPA above the minimum for his program (or any program for that matter). Rather than transferring to another degree such a student is likely to be forced out of the university entirely in short order.

> but at least at US universities most higher level courses are part of a sequence and are thus only offered at a specific time of year. Thus failing delays your degree by a year and creates all sorts of logistical issues for the student.

This often also holds in Germany. And indeed if you thus fail an exam, your degree is delayed and you might have logistical issues.

The moral of this: learn hard so that you don't fail exams. :-)

A complete ban feels like overkill. A cooldown period of X years would give someone time to reconsider and grow as a person.

Granted, this is without consider the financial side of things.

> where all you got was the outline of the island

As a blind person I can't help but notice that this "innovative" question design is inherently inaccessible to people like me. Yes, I know I'm not the norm. However, I can't help but notice that this avoidance of text based assignments is making accessible teaching even more impossible. Say welcome to a new generation of Digital Divide.

It’d be bad if there was no alternative. But different people can get different assignments and that’s ok.

Complaining about this is like me complaining about people drinking milk since I’m lactose intolerant.

> It’d be bad if there was no alternative. But different people can get different assignments and that’s ok.

While there are possibly noble goals behind your suggestion, in practise this puts anyone outside the mainstream in the category of “other,” people to be managed separately. I’ll leave to your imagination how much work is often put into supporting these “other” assignments and how up-to-date they’re kept va the mainstream.

> Complaining about this is like me complaining about people drinking milk since I’m lactose intolerant.

If this is genuinely your approach, you are being part of the problem; if taking a step back to reassess feels like too much work, I’d encourage you to explore why it feels that way, what emotions is this bringing up internally?

I think "everyone gets the exact same test" is a whole level of hell worse than "everyone gets a test that makes sense for their abilities and goals"

Are you just concerned about the administration side of things?

I think the GP here has a pretty reasonable outlook here

"Some people can't do some things the same way as others and that's ok, people are different"

Is the vibe I got from that post

Either the test must be the same for everybody or it is not a meaningful signal.

Yes. But there will always be an obscure case you can’t cover. Let’s say a test is written. Someone can’t write, but can dictate answers. Is it still the same test or not?

On the other hand, how far can society tip-tap around such alternative takes? Could he go study in a field where hand writing is expected and society would still be forced to adapt?

> in practise this puts anyone outside the mainstream in the category of “other,” people to be managed separately.

So what ? This is literally the case. The commenter has to be managed separately

The commenter (rightly or wrongly) is -I think- suggesting that assignments etc should be from the outset be designed to be accessible in their original form for everyone, and not just "okay, this works for 90% of people, lets worry about the rest separately".

An example I have personally is when designing board games I now have internalised to never just use colour to communicate anything. Never solid colour which can be difficult to determine by the colour-blind, but different colours have different stripes or patterns embedded in them. So if you can't see the colour, you can recognise the stripe spacing, if you can't see the stripe spacing due to low visual-acuity, you can see the colour... and ideally different textures for the blind. (not pratical as it would wear off in the shuffling, but the idea is there).

But an assigment cannot be accessible to everyone by definition. You can't have a vision-based test accessible to a blind person as you can't have a sound-based test accessible to a deaf person.

So I'm not a disability advocate, but I did date someone who was so I was immersed in it, but never having studied it so take my answer as a best effort trying to piece together from memories.

I think the answer would be to make assignments multi-faceted that can be approached differently. Like instead of having students write a report on book they read, which priveledges the those with sight, non-dyselxics, and those that can type let them perhaps record a podcast, film a YouTube video, draw a webcomic etc that lets students show what they know. And then reflect on why does it have to be a book that's read? It could be a film, a mini-series, a radio-play, a graphic novel, a play. That allows people to approach it using their abilities instead of being hampered by disabilities such as blindness, deafness, ADHD, dyslexia, being non-verbal, etc.

It means you've distilled the assignment to it's core: can you summarise the plot, identify key moments, recognise themes and metaphors, and place the work within the historical context in which it was produced.

There you have an assignment that is much more accessible.

The tricky part is that marking them objectively is much more difficult because the criteria of evaluation are not necessarily monolithic but need to take into account the student and the mediums chosen.

...but it's still a single accessible assignment.

It's still separate assignments. The only difference is wether you throw them into a single meta-assignment. Or a separate one with main + fallbacks.

The problem with such assignment is it's very hard to balance the alternatives to be equal. Reading a book takes quite a bit longer than watching a film or even mini-series. IMO it's perfectly fine for someone blind to watch a film and then make a report on it. But how many healthy people would just go for a film since it's quicker? And reading a book is quite a bit different experience than a film due to much more space for imagination and interpretation in a book.

P.S. calling people with no disabilities „privileged“ is just... wrong. It's not a privilege to have a set of eyes/hands/etc.

> If this is genuinely your approach, you are being part of the problem; if taking a step back to reassess feels like too much work, I’d encourage you to explore why it feels that way, what emotions is this bringing up internally?

How am I part of the problem? Do you mean I should make a fuss if I get limited options? Or have emotions towards what... My genetical makeup? All in all, I've zero emotions about my lactose intolerance. Being mindful about what I eat is just natural for me. Just like for other people with food allergies/intolerances.

> in practise this puts anyone outside the mainstream in the category of “other,” people to be managed separately

Is your goal to get everyone down to the same level by removing all learning material that may be inaccessible to somebody?

Should people just stop consuming anything with lactose to make it more universal for people like me? But then a friend of mine will ask to stop eating fish. Someone else will take away chocolate. At the end of the day all of us will be much worse off.

What would you need to tackle this assignment?

I think you're right to complain about the design of the question. It used Google's disadvantage in that it couldn't process visual information to add difficulty to the task.

However I'm pretty sure there's many of these challenges in an average stem subject. An example would be information dense diagrams which describe some chemical process.

I'm genuinely curious to understand how someone manages to understand these without sight. On my side, I can barely visualize so it was always extremely hard to decipher them and even worse remember them.

Maybe they could 3d print the outline to make it more accessible? Or perhaps the goal is to use the image file with the outline to find the island, something that could/should be achieved via code?

I wouldn't be surprised if a novel solution to the problem emerged from students with diverse abilities. I don't think that brute forcing the problem by comparing the image of every island on the planet vs the outline visually is the best path forward. Its probably something like, create a ratio of the size of each of the bays and then probe for values that fit those quantities on wikipedia.

As a university professor, I admit with some shame that accessibility issues for specific problem types is not on my radar. “Innovation” isn’t the main culprit here.

Fortunately, my university has a good accessibility center that takes care of accommodation issues (large print versions of tests, etc.). I just send them my tests and they take care of it. It’s a great service, and absolutely crucial because I simply don’t have the time to customize assessments. I assume they would get in touch if they were unable to retrofit accessibility onto an assessment, but that hasn’t happened in my fifteen years of employment.

The good news is that avoiding text-based assignments isn't likely to last. Multimodal agents should be more than capable of identifying islands by shape, or will be soon.

I gave Gemini a mockup of a UI and it generated the stupid thing perfectly in code and CSS. Recognizing an island seems trivial in comparison.

Yes, it's true that the one example I presented from a quiz game I played at a conference 10+ years ago was not accessible to some people. If I could remember the other questions they used in that quiz I would share them here, but that's the only one that stuck in my brain.

Among others, Howard Rheingold has been active in this space. For those interested, check out the Peeragogy Handbook and the post that sparked the idea[0].

> The more I give my teacher-power to students and encourage them to take more responsibility for their own learning, the more they show me how to redesign my ways of teaching

[0]: https://clalliance.org/blog/toward-peeragogy/

It's fun to see how old that article is and that the ideas still apply! Power dynamics are not considered enough when people talk about education. My believe is that the more you balance the power dynamic, the more learning is prioritized over education.

For an even earlier account, see the learning networks described by Illich (1971) in the Deschooling Society [0].

[0]: https://en.m.wikipedia.org/wiki/Deschooling_Society

https://bsky.app/profile/hrheingold.bsky.social/post/3lprsqu... points to a compiled syllabus and links to recordings of lectures and video chats in a free pdf, about an "intro to cooperation studies": https://rheingold.com/texts/IntroToCooperationStudies.pdf

> Designing "ChatGPT-proof assignments" feels to me like a similar level of intellectual challenge.

This is indeed true. But a challenge is that few professors are being given the time and training to help do this. When you are on a 4/4 and just keeping your head above water you don't tend to have enough time to adopt experimental pedagogies and completely replan your courses. And professors are largely being tasked with doing this independently rather than having universities offer training or support with adjusting methods to be more AI resistant. And unfortunately the fast speed of development of these tools is making good ideas obsolete quickly. I know some people who switch to having students make podcasts rather than writing papers as a final assignment and then we started seeing "create your own podcast" tools appear and made it roughly as easy to cheat on this assignment as a traditional paper.

My suggestion here isn't to have the professors do this: it's for the professors to challenge their students to do this.

I am friends with an unusually large number of professors, many of which are at very top institutions. The "make your own pedagogy" approach for students leaves huge portions of the classroom doing zero work whatsoever.

Thats useful, thanks. Sounds like this approach won't help most students.

What was the point of having those students in the classroom in the first place?

Taking their (or their parent's) money of course

alternative strategies will have them doing work and learning

And then forgetting. If you're not interested, you cram for the test and then forget. I see little point in forcing people who don't want to learn into classrooms. Worth a read https://www.astralcodexten.com/p/a-theoretical-case-against-...

Interesting essay. Makes some very good points, however I have to disagree with this one.

> ... they keep some sort of overall concept of learning. This is a pretty god-of-the-gaps-ish hypothesis, and counterbalanced by ...

The author is really missing the obvious here. Learning difficult subjects fundamentally changes how you think about and approach things. People aren't born able to engage in critical thinking or being able to reason algebraically or with an ability to navigate formal logic. That doesn't mean school is the only way to impart such skills, but it is certainly one of the ways.

Of course if the metric you use is "ability to answer trivia" then you are going to fail to capture that aspect.

> Learning difficult subjects fundamentally changes how you think about and approach things.

Citation needed on this. But even if I accept the premise, what percentage of school is leaning difficult subjects? For me it was <5% and the other 95+% was stuff that could be beaten easily with rote memorization. The only classes in that 5% were upper level courses that only students who want to be there take anyway.

Is it not self evident? A subject doesn't have to be particularly difficult to impact how you think. Being exposed to algebra provides you with new ways of looking at the relations between things. Being exposed to even basic world history gives you perspective that you wouldn't otherwise have had about society at large. I've observed the effect in both myself and those around me to the extent that it appears self evident to me.

Unfortunately any of the obvious metrics you might use to quantify this would seem to be hopelessly biased. The more advanced the degree someone holds the farther above average that individual tends to be in various ways. I doubt it would be possible to control for such large cross correlations.

Perhaps asking people who learn multiple languages could provide subjective evidence since that doesn't have nearly as much of a correlation with other abilities.

There really are students that will work hard when given direction but will flounder when asked to fully self direct.

How does making podcasts solve anything? Students can just read out what the AI writes into an audio recording.

Why is this problem being fretted over? In-class written and oral tests should be fine to assess students. If AI helps them learn or even cram the material, great!

One of the skills that students are typically terrible at is convincingly recreating the flow of a conversation that is written down. Shakespeare, spoken in meter-less monotone with none of the prosody that indicates actual speech by a human being communicating something; Prosody you can often decode even in a language you have no knowledge of. If they manage to fake it convincingly for an hour-long podcast, they have taught themselves acting on a level that is commercially useful and honestly understand much of what is being said.

If they are just working from AI notes and improvising the conversation, arguably they are simply doing the work.

What I don't know is whether an instructor would be allowed to earnestly grade an hour-long podcast subjectively.

Homework and/or project assignments are often a major component of scoring for many courses.

You can do different podcast structures than a single person reading a script, which can produce some useful social barriers to cheating.

A lot of kids also suck at cheating so the small barrier of "ask ai to generate a script and then read it" appears to stop a bunch of cheaters.

There are limitations to timed evaluations such that oral exams or blue books are not panaceas.

>on a 4/4

That's teaching four distinct courses?

A 4/4 means that you teach four classes each semester. This is a pretty typical schedule for non tenure track or adjunct professors in the humanities or even tenure track faculty at slacs, though I've seen 5/5s before. It is a lot of work to update this many classes to adapt to AI.

That seems like a crazy amount of work. Four classes, or four sections of the same class?

Depends, but it is often four distinct classes. And yeah, it is a lot of work. This is why reusing preps is so valuable and why it is a bit ridiculous to see people demanding that professors suddenly figure out (on their own) how to rewrite their syllabi to mitigate cheating with AI.

Yikes.

My professor often taught classes with a couple sections, I’d help out sometimes—even that was a ton of work, pretty hard to do the main job (research) during those semesters. I wonder if (given the demographics of the board) folks are suggesting things from a similar place of informed ignorance to me—coming from research oriented STEM universities. In that case it could be reasonable to say “well, the situation with our classes has gotten so dire with AI, it might make sense to sacrifice a semester of research to sort that all out.” Of course if you are already prepping for four classes, there is not much slack to sacrifice…

In that case a lot of us would be specifically more wrong than a random person plucked off the street, somebody who thinks the main job of every professor is teaching.

If a teacher knows the topic that they're teaching, then a 30 second talk with a student is enough to determine if the student actually knows anything. Maybe "assignments" aren't the best method of building and verifying knowledge.

As a professor of Computer Science, I'm about ready to allow my students to use AI -- if you can't beat 'em, join 'em. But they're not going to like it, because what I'm going to do is significantly raise the bar.

For the last several years I've been allowing students (college sophomores) to do take home exams where they essentially have to build a networked system in C++. I tell them not to use AI but I have to assume they are.

Even still, the solutions are not all that great, despite unlimited time and resources, they still submit work that falls into a roughly B average. Which, if they didn't have AI, maybe that would be a C average, so perhaps the standards just need to go up. Still, it's not like they're all getting 100%.

AI is still really only as good as the person driving it, and it still, despite all the hype, hallucinates like crazy, such that if they don't pay close attention and constantly course correct, they can't expect the project to actually work.

I find it difficult to think of examples that aren't just based on niche information (what did we talk about on this day? what analogy does the teacher make repeatedly?). Maybe multimodal stuff?

Even multimodal stuff is something is advancing pretty quickly. And with the more widespread incorporation of things like this, the information will be fed back into AI systems as training data making finding new things more difficult.

That comment struck me as odd if only because Plato is not commonly part of the standard critical thinking curriculum. Introduction to philosophy or history of philosophy yes, but critical thinking no.

This is not to say that Plato didn't think critically. Of course he did, as do all (or most) philosophers. I'm just talking about university courses and textbooks.

Yeah, that does sound like a good idea and a good way to prove one has learned the ideas presented in the course and the relationship between them well

“Cannot be done with chat gpt”

Is a military base the only place “no devices” can be enforced or something? How deep is this addiction? I’m scared to ask what the academic version of fizz-buzz would be at the end of a 4 year degree, “hey write one paragraph describing a simple paradox, an example of irony, or an example of a metaphor”.

>Is a military base the only place “no devices” can be enforced or something?

I mean with the force of law, yea. Businesses get away with a little bit more in places because they pay you to show up.

But if you think anyone is going to university/college to have all their devices taken away all the time and pay for the privilege then you might be confused.

No but people do pay universities for (among other things) a meaningful credential that testifies to their knowledge and abilities.

If taking away phones is what’s required to make that credential meaningful again, then the universities must do it.

I think of it like a personal trainer. Do you pay them to make you sweaty and sore? Not really, but sweat and soreness are consequences of doing what it takes to build muscle/endurance/etc.

>meaningful credential that testifies to their knowledge and abilities.

No they don't, or at least a significant percentage of them don't. They pay the university because it is a paywall between them and even getting an interview in the first place.

But that isnt a good long term strategy. If degrees hold no real signal, employers are going to start noticing and the value of a degree decreases.

So its in universities best interest to actually test for competence.

... In theory.

Are people really this craven? I went to college because I like learning stuff. I literally never once considered work even as I went on to grad school.

I mean, before you get to military bases, there are Pearson's testing centers that have lockers for your phone/gear and proctored exams.

How broadly do you propose to enforce such a ban? Most students do their homework at home so I'm not sure how you're imagining this would work.

Even for exams I had my cellphone and backpack on me. You just weren't allowed into them. The only exception to that in my experience was exams proctored in a dedicated testing center where there are lockers, multiple human observers, and lots of cameras.

Reminds me of https://xkcd.com/810/

I am a zoomer. Is it really that hard for everyone to NOT use AI? Using AI very clearly atrophies your cognitive skills.

I'm a millennial who has watched the complete failure of my generation to realize the promise of those Obama 'Hope' posters.

"Everyone don't do X" runs into problems when there are clear incentives to do X. Like, say, grades which eventually will be used to determine who gets to graduate, who gets into which schools, who ultimately gets easier access to higher paying jobs. And simple convenience is an incentive in itself - if doing X saves time, it will be the default.

This same dynamic is why we still have a climate crisis, even though everyone has known about the problem for thirty years.

"Everyone just do Y" is what we call a collective action problem. When there are clear incentives to defect, it is also a free rider problem.

Reply on news.ycombinator.com