ChatGPT gives me a summary of the article in The Guardian: https://www.theguardian.com/commentisfree/2024/feb/13/software-student-cheated-combat-ai
Robert Topinka, a lecturer at Birkbeck, University of London, explores the dilemma professors face with students’ use of AI to write essays. After flagging an essay as “100% AI-generated,” Topinka lands in a tough spot when an exceptionally bright student challenges the accusation. The case highlights the problems with AI detectors like Turnitin, which can mistake legitimate technological support used by students for cheating. Topinka argues we must adapt academic assessment to the AI era, proposing alternatives such as presentations and podcasts to demonstrate students’ critical and original thinking, avoiding unfair accusations and promoting equal educational opportunities.
Over the past year, I’ve given courses to lecturers from more than seven universities (you can see part of the content at https://wasabi.essi.upc.edu/ludo/cursos), and I always get questions about how to detect whether a piece of work was done by AI instead of a person.
Copied work is nothing new. When I was still a student, I tutored a course called Methodical Programming at a cram school, and one day they asked me to solve an exercise that turned out to be the same one that appeared on the exam. Don’t ask me where that problem statement came from, because I don’t know. I didn’t keep working at that school to find out.
Since the web has existed, we’ve had things like “El Rincón del Vago,” where students share notes, exercises, and solved assignments. And on Instagram, students see ads for companies that offer to do their bachelor’s or master’s theses… so they can spend more time on their crush. I myself get ads on X (formerly Twitter) from companies that offer to write research papers in my field and publish them under my name in prestigious journals.
But AI is the problem. Sure. Because it lets you do things at scale and at negligible cost. And now we’re all running to it.
And what we want are easy fixes, like taking anti-inflammatories, but applied to teaching quality and to the peace of mind that a dashboard gives you. Systems like Urkund and Turnitin integrate with our Moodles and Sakais and tell us what percentage of our students’ submissions are original or are copies of things online or of their classmates’ work.
As Bruce Willis said in “The Kid”: “sandwiches give you security.” And the green color next to a submission gives you peace of mind.
But students have been inserting Cyrillic characters that look like Latin letters for years now (see the table of confusable characters at https://github.com/mindcrypt/uriDeep/blob/master/data/deepDiccConfusables.txt), and they get around anti-plagiarism systems without a problem.
And what about systems that detect AI-generated content?
Back in February 2023 I already said—and it’s on YouTube, etched in silicon—that for texts under a page it’s impossible to say with certainty whether a text was generated by AI or written by a person. Later, researchers at the University of Maryland published “Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-generated text be reliably detected? arXiv:2303.11156 [cs.CL]. https://doi.org/10.48550/arXiv.2303.11156”
We read in the abstract:
Next, we provide a theoretical impossibility result indicating that as language models become more sophisticated and better at emulating human text, the performance of the best possible detector declines. For a sufficiently advanced language model aiming to imitate human text, even the best possible detector could perform only marginally better than a random classifier. Our result is general enough to capture specific scenarios such as particular writing styles, clever prompt design, or text paraphrasing.
In other words, NO, you cannot reliably detect whether a text was generated by AI. The system won’t be reliable and will produce errors: false negatives and false positives.
Now, let’s talk ethics. Real AI ethics, not the sham kind. The ethics of a professor who decides to outsource their judgment to a machine that says a student “cheated with AI.” The ethics of a company that sells the service of that machine that… DOES NOT WORK.
False positives can affect a student’s career, or get a researcher blacklisted by a publisher based on an AI discriminator.
“AI has told me you cheated with AI.” That’s a display of cynicism and laziness, of not wanting to do things properly. Of not stopping to think that if an AI can do the exercise you’re asking for, maybe you should ask for something else.
More than one lecturer has told me, to my horror, that her detector for AI-generated content is ChatGPT. And that she believes it. For a few months now, ChatGPT has stopped sticking its neck out when asked whether a text was generated by AI.