Cambridge University Press – Does ChatGPT make the Grade?

Cambridge researchers discovered indicators of ChatGPT’s default writing style in student essays. Despite the small sample size, the research represents an initial attempt to figure out how students use ChatGPT in essay writing, and how ChatGPT-assisted essays perform in assessment.

Researchers noticed some tell-tale signs of ChatGPT’s writing style, such as tautology, redundancy and excessive use of words like ‘however’.

They also found that ChatGPT-assisted essays performed worse than expected in skills like analysis and comparison.

In the research, there were three participants who were undergraduate students, and each wrote two essays with the help of ChatGPT. The rules require users to be 18 and above. These essays were evaluated by examiners and compared to 164 essays written by genuine IGCSE students. The undergraduates were then interviewed, and their essays were examined.

Researchers discovered that ChatGPT-assisted essays excelled in ‘information’ and ‘reflection’ compared to non-ChatGPT-assisted essays written by Cambridge IGCSE students (aged 14 to 16) but performed below expectation in ‘analysis’ and ‘comparison’.

The researchers pointed out that ChatGPT’s default writing style “echoes the bland, clipped, and objective style that characterises most generic journalistic writing found on the internet”.

• The use of Latinate vocabulary (multisyllable, sophisticated vocabulary above the expected level of study)
• Paragraphs starting with specific discourse markers (‘however’, ‘moreover’, and ‘overall’ — followed by a comma)
• Numbered lists with items followed by colons
• Pleonasms (using unnecessary words to convey meaning e.g. ‘true fact’ or ‘free gift’)
• Tautology (saying the same thing twice e.g. ‘we must come together to unite’)
• Repetition of words or phrases and ideas
• Consistent use of Oxford commas (e.g. ChatGPT has many uses for teaching, learning at home, revision, and assessment)

In interviews, the researchers also found commonalities in how participating students used ChatGPT to different extents. Patterns emerged, with a wide consensus among students that ChatGPT is useful for gathering information quickly and can be integrated into essay-writing through steps like specific enquiries about topics, essay structuring and writing the essay.

However, the students also considered that complete dependence on ChatGPT would produce essays of poor academic quality.

“Our findings offer insights into the expanding domain of generative Artificial Intelligence (AI) and assessment, which remains largely unexplored.”

Jude Brady also noted that “Although our sample size is small, these discoveries are promising as they have the potential to assist and guide teachers as well as students. We advocate for further research with more representative and larger sample sizes. We hope our research helps in recognising text generated by ChatGPT.”

“For students and the wider population, learning to use and detect generative AI is becoming an important aspect of digital literacy.”