A Survey Study on the State of the Art of Programming Exercise Generation Using Large Language Models

This paper analyzes Large Language Models (LLMs) with regard to their programming exercise generation capabilities. Through a survey study, we defined the state of the art, extracted their strengths and weaknesses and finally proposed an evaluation matrix, helping researchers and educators to decide...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings / Conference on Software Engineering Education & Training pp. 1 - 5
Main Authors: Frankford, Eduard, Hohn, Ingo, Sauerwein, Clemens, Breu, Ruth
Format: Conference Proceeding
Language:English
Published: IEEE 29.07.2024
Subjects:
Artificial Intelligence
ChatGPT
Data privacy
Education
Fitting
Large language models
Navigation
Programming Education
Programming Exercise Generation
Programming Exercise Generation Benchmark
Surveys
Transforms
ISSN:2377-570X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper analyzes Large Language Models (LLMs) with regard to their programming exercise generation capabilities. Through a survey study, we defined the state of the art, extracted their strengths and weaknesses and finally proposed an evaluation matrix, helping researchers and educators to decide which LLM is the best fitting for the programming exercise generation use case. We also found that multiple LLMs are capable of producing useful program-ming exercises. Nevertheless, there exist challenges like the ease with which LLMs might solve exercises generated by LLMs. This paper contributes to the ongoing discourse on the integration of LLMs in education.
ISSN:2377-570X
DOI:10.1109/CSEET62301.2024.10662990