Delving into the psychology of Machines: Exploring the structure of self-regulated learning via LLM-generated survey responses

Large language models (LLMs) offer the potential to simulate human-like responses and behaviors, creating new opportunities for psychological science. In the context of self-regulated learning (SRL), if LLMs can reliably simulate survey responses at scale and speed, they could be used to test interv...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Computers in human behavior Ročník 173; s. 108769
Hlavní autori: V.D.E. Vogelsmeier, Leonie, Oliveira, Eduardo, Misiejuk, Kamila, López-Pernas, Sonsoles, Saqr, Mohammed
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier Ltd 01.12.2025
Predmet:
Artificial intelligence
Educational psychology
Psychometric validation
Self-regulated learning
Survey simulation
Survey simulation
Self-regulated learning
Educational psychology
Artificial intelligence
Psychometric validation
ISSN:0747-5632
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Large language models (LLMs) offer the potential to simulate human-like responses and behaviors, creating new opportunities for psychological science. In the context of self-regulated learning (SRL), if LLMs can reliably simulate survey responses at scale and speed, they could be used to test intervention scenarios, refine theoretical models, augment sparse datasets, and represent hard-to-reach populations. However, the validity of LLM-generated survey responses remains uncertain, with limited research focused on SRL and existing studies beyond SRL yielding mixed results. Therefore, in this study, we examined LLM-generated responses to the 44-item Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich & De Groot, 1990), a widely used instrument assessing students’ learning strategies and academic motivation. Particularly, we used the LLMs GPT-4o, Claude 3.7 Sonnet, Gemini 2 Flash, LLaMA 3.1–8B, and Mistral Large. We analyzed item distributions, the psychological network of the theoretical SRL dimensions, and psychometric validity based on the latent factor structure. Our results suggest that Gemini 2 Flash was the most promising LLM, showing considerable sampling variability and producing plausible underlying dimensions and theoretical relationships that are partly aligned with prior theory and empirical findings. At the same time, we observed discrepancies and limitations, underscoring both the potential and current constraints of using LLMs for simulating psychological survey data and applying it in educational contexts. •We simulated MSLQ responses using five LLMs: GPT-4o, Claude 3.7 Sonnet, Gemini 2 Flash, LLaMA 3.1–8B, and Mistral Large.•Psychometric analyses revealed similarity and inconsistencies with theory and prior empirical results.•Gemini 2 Flash produced outputs most consistent with SRL theory.•Our study contributes to growing conversations on the use of synthetic data in psychology and education.•Our study also raises critical questions about over-coherence, variability, and construct fidelity in LLM-generated data.
ISSN:0747-5632
DOI:10.1016/j.chb.2025.108769