Evaluating the effects of Generative AI on student learning outcomes: Insights from a meta-analysis

The emergence of Generative AI technologies, represented by ChatGPT, has triggered extensive discussions among scholars in the education sector. While relevant research continues to grow, there is a lack of comprehensive understanding that systematically measures the effects of Generative AI on stud...

Full description

Saved in:
Bibliographic Details
Published in:Educational Technology ＆ Society Vol. 28; no. 3; pp. 226 - 240
Main Authors: Hu, De-Xin, Pang, Dan-Dan, Xing, Zhe
Format: Journal Article
Language:English
Published: International Forum of Educational Technology ＆ Society 01.07.2025
International Forum of Educational Technology & Society, National Taiwan Normal University, Taiwan
International Forum of Educational Technology & Society
Subjects:
Analysis
Education
Educational evaluation
generative ai
generative artificial intelligence
Generative artificial intelligence in education: Theories, technologies, and applications
Learning
learning outcomes
meta-analysis
Methods
Study and teaching
Generative AI
Learning outcomes
Education
Generative artificial intelligence
Meta-analysis
ISSN:1176-3647, 1436-4522, 1436-4522
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The emergence of Generative AI technologies, represented by ChatGPT, has triggered extensive discussions among scholars in the education sector. While relevant research continues to grow, there is a lack of comprehensive understanding that systematically measures the effects of Generative AI on student learning outcomes. This study employs meta-analysis to integrate findings from previous experimental and quasi-experimental research to evaluate the impact of Generative AI on student learning outcomes. The analysis of 44 effect sizes from 21 independent studies indicates that Generative AI tools, compared to traditional AI tools or no intervention, moderately enhance student learning outcomes (g = 0.572). These tools significantly improve the cognitive (g = 0.604), behavioral (g = 0.698), and affective (g = 0.478) dimensions of learning outcomes. In addition, the study identifies and examines 6 potential moderating variables: educational level, sample size, subject area, teaching model, intervention duration, and assessment instrument. The results of the moderating effects test reveal that sample size and assessment instrument significantly influence the effectiveness of Generative AI. For sample size, the effect of Generative AI on small samples (g = 1.216) is greater than that on medium (g = 0.476) and large samples (g = 0.547). For assessment instrument, the effect of Generative AI on self-developed tests (g = 0.984) is greater than that on standardized tests (g = 0.557). The meta-analysis result indicated that the use of Generative AI should be supplemented with detailed guidance and flexible strategies. Specific recommendations for future research and practical implementations of Generative AI in education are discussed.
ISSN:1176-3647
1436-4522
1436-4522
DOI:10.30191/ETS.202507_28(3).TP02