Delving into the psychology of Machines: Exploring the structure of self-regulated learning via LLM-generated survey responses - V.D.E. Vogelsmeier, Leonie; Oliveira, Eduardo; Misiejuk, Kamila; López-Pernas, Sonsoles; Saqr, Mohammed (Computers in human behavior, 2025)
Large language models (LLMs) offer the potential to simulate human-like responses and behaviors, creating new opportunities for psychological science. In the context of self-regulated learning (SRL), if LLMs can reliably simulate survey responses at scale and speed, they could be used to test intervention scenarios, refine theoretical models, augment sparse datasets, and represent hard-to-reach populations. However, the validity of LLM-generated survey responses remains uncertain, with limited research focused on SRL and existing studies beyond SRL yielding mixed results. Therefore, in this study, we examined LLM-generated responses to the 44-item Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich & De Groot, 1990), a widely used instrument assessing students’ learning strategies and academic motivation. Particularly, we used the LLMs GPT-4o, Claude 3.7 Sonnet, Gemini 2 Flash, LLaMA 3.1–8B, and Mistral Large. We analyzed item distributions, the psychological network of the theoretical SRL dimensions, and psychometric validity based on the latent factor structure. Our results suggest that Gemini 2 Flash was the most promising LLM, showing considerable sampling variability and producing plausible underlying dimensions and theoretical relationships that are partly aligned with prior theory and empirical findings. At the same time, we observed discrepancies and limitations, underscoring both the potential and current constraints of using LLMs for simulating psychological survey data and applying it in educational contexts. •We simulated MSLQ responses using five LLMs: GPT-4o, Claude 3.7 Sonnet, Gemini 2 Flash, LLaMA 3.1–8B, and Mistral Large.•Psychometric analyses revealed similarity and inconsistencies with theory and prior empirical results.•Gemini 2 Flash produced outputs most consistent with SRL theory.•Our study contributes to growing conversations on the use of synthetic data in psychology and education.•Our study also raises critical questions about over-coherence, variability, and construct fidelity in LLM-generated data.