Professor: "The website said you used an AI." Student: "Well, here's a website claiming I didn't use an AI." Professor: "I don't know if I believe that website, let's look into the merits of how it works." Student: "Yes, let's do that." Professor: ...
I'm trying to think if you can indeed do something with a fair test.
I don't know, that might be incredibly difficult forever, or it might be something that someone has a breakthrough in and manages to do.
Perhaps it's only useful for particularly egregious examples, and wouldn't be able to detect an AI text that was subsequently edited. That would be fine. When I was a TA in college, we ran students' papers through a plagiarism detector, which would spit out some % number based on what it thought was the amount/likelihood of plagiarism. The number itself was bunk, but across a class of 30 people, if one or two papers had much higher numbers than the other 28, I could click and look at it's justification. Sometimes it was a meaningless false positive, other times it was "here are the paragraphs that were copied verbatim from Wikipedia".
One major issue is the feedback loop: There is lots of entrophy to hide a signal in regular English (whether you follow the methods of Stylometry or Steganography), and even an unmodified AI model has a stylometric signature. But if a significant portion of text we consume ends up being written by a handful of AI models, over time people will adopt many of these features. This is obvious right now, with ChatGPT having a certain writing style and some people unconsciously copying that style, and it will remain true no matter how subtle these things become.
well, why not train an AI on it?
I love this piece of history so much, thank you for sharing it! Adding this one to my bookmarks :-)
“The Waag is still open as a tourist attraction, and official certificates are available.”
Maybe with modern science we can make someone much lighter than they ought to be. (Put helium in their stomach?) Then they would be forced to issue a certificate that said that they could not guarantee that the person is not a witch :D
If universities offer a learning environment that actually provides the students better learning than what they can get at home, the pieces of paper they hand out will continue to mean something. If they don't, they were already on a downward spiral kicked loose by online learning platforms handing out certificates.
It's harder but probably not that hard.