Can ChatGPT Analyze Textual Data? The Case of Conceptual Metaphors in Short Stories of Language Assessment

ChatGPT, a modern artificial intelligence (AI) chatbot, has emerged as an unprecedented breakthrough in multiple domains traditionally dominated by humans. Its ability to engage in human-like conversations has the potential to influence the fields of linguistics and education. The fundamental functi...

Full description

Saved in:
Bibliographic Details
Published in:Journal of language teaching and research Vol. 16; no. 5; pp. 1665 - 1672
Main Authors: Geng, Hui, Wei, Han, Nimehchisalem, Vahid, Azar, Ali Sorayyaei
Format: Journal Article
Language:English
Published: Academy Publication Co., LTD 01.09.2025
Subjects:
Analysis
Computational linguistics
Education
Language processing
Methods
Natural language interfaces
China
Malaysia
ISSN:1798-4769, 2053-0684
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:ChatGPT, a modern artificial intelligence (AI) chatbot, has emerged as an unprecedented breakthrough in multiple domains traditionally dominated by humans. Its ability to engage in human-like conversations has the potential to influence the fields of linguistics and education. The fundamental functions of ChatGPT in teaching and learning have been the subject of some research, but its application in textual analysis has received scant attention. This study aims to investigate how ChatGPT assists in analyzing conceptual metaphors (CMs) in short stories used in language assessment. Based on the Conceptual Metaphor Theory (CMT) by Lakoff and Johnson (1980), the study identified the structural, orientational, and ontological metaphors in 22 short stories from the book Tests and Us 2, first by the cutting-edge AI program ChatGPT (GPT-4), then refined by the researchers and validated by linguistic experts. The results showed a total of 250 conceptual metaphors, including 131 structural metaphors, 64 ontological metaphors, and 55 orientational metaphors. When validated by human specialists, GPT-4 accurately recognized conceptual metaphors in 81.2% of the cases, amounting to 203 instances. The most dominant error made by GPT-4 was classifying non-metaphoric expressions as metaphoric, followed by providing unclear explanations and classifying metaphoric expressions as non-metaphoric. Errors associated with being too general and too non-literal, unmatched categorization as well as wrong mapping of source or target domain also occurred. Our study shows that ChatGPT, despite its controversial position in academic settings, can be used as a relatively reliable tool in aiding the analysis of textual data.
ISSN:1798-4769
2053-0684
DOI:10.17507/jltr.1605.24