ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot developed by OpenAI. ChatGPT currently can communicate with clients through text conservations and can execute complicated language tasks, including summarization. Alternatives to ChatGPT include Perplexity.AI and Quora Poe. These bots can facilitate teaching, learning and assessment in the long term. Meanwhile, in this blog post, we mainly discuss on:
- How to identify essays through detection tools or other mechanisms, and
- How to design ChatGPT-resilient essay assignments and other assessments.
Identifying essays that ChatGPT creates
It is always challenging to detect essays that ChatGPT creates. Since ChatGPT created different responses every time, even with the same prompt, it is difficult for “Turnitin Similarity” to detect contents created by ChatGPT. As told by colleagues from Turnitin, Turnitin is currently developing tools for detection.
There are existing detection tools, including GPTZero, Open AI Text Classifier and Writer AI Content Detector. These detectors often detect the perplexity and burstiness of the article to identify if the article is created by humans. You can also simply import the article to ChatGPT to calculate the perplexity and burstiness of the article. These tools are user-friendly, but these detectors often cannot provide a correct judgement in our testing.
If multiple essays from the same students are provided, we can check whether the writing style across essays is similar. Teachers can analyse manually or through computational linguistics tools like Coh-Metrix. For example, students often write essays with American or British spelling only, and with a similar standard deviation of the mean length of paragraphs and the mean length of sentences as well as referential cohesion (and some other computational linguistics metrics).
Besides analysing the content and writing style, we can also examine the metadata of the essay submission files. If students submit the essay through MS Word files, you can check if there are abnormalities in the properties of the essay file. If students used MS Word throughout the writing process, there should be several revisions, and the date created and modified should not be too close. Students can also provide a version history if students used Google Docs for writing.
Designing ChatGPT-resilient essay assignments
Besides analysing the content of the essay, you can also identify essays by analysing the writing process. For example, you can ask students to:
- Submit draft notes or a writing plan in the early stage of the assignment (and compare them with the final essay)
- Record the writing process through Zoom screen sharing
- Reflect on the writing process and evaluate the meta-cognitive process of the student
Teachers can also ask students to write multiple essays such that we can examine if the writing style is different across essays.
As reported in recent studies published on arXiv (Study 1 and Study 2), ChatGPT currently is still not perfect in some reasoning skills, including inductive reasoning, non-textual semantic reasoning (spatial reasoning, mathematical reasoning), multi-hop reasoning, physical reasoning and psychological reasoning. Teachers can consider designing essay questions that partially assess the abovementioned reasoning skills. Teachers can read these studies for more details and get inspired to create appropriate essay questions. For example,
- Teachers can think about designing assignments that require explanation through infographics, figures and maps.
- Teachers can add atypical requirements that systems often overlook, such as “In the third sentence of your response, the first word should be started with the letter “C” and one of the words should be started with the letter “V” “.
- Teachers can ask students to provide more references to support their arguments.
These practices can increase the hurdles of directly copying content generated from AI tools as answers.
Teachers are also encouraged to assess students in multiple ways and examine whether students’ performance in essay assessments aligns with other assessments.
Using ChatGPT ethically in assignments and T&L
If you allow students to use ChatGPT in your assignments, besides providing proper citations, you can consider asking students to write an additional reflection on the following:
- The comparison between human development and AI-facilitated development
- Their experience in using ChatGPT to speed up the development process
- Current limitations and how the AI-facilitated development process could be improved.
You can find more ideas from Dr Ethan Mollick from the University of Pennsylvania and Dr Sean McMinn from The Hong Kong University of Science and Technology on using AI tools to facilitate teaching, learning and assessments. Other interesting ChatGPT prompts can be found online.
Author: Dr Leon Lei
The author has used Grammarly to polish the article, but has not used ChatGPT in the writing process. The recording of the writing process can be found at https://youtu.be/1eD86rorzCA. The article may be further updated without notice.
