Tackling Students' Coding Assignments with LLMs

State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating o...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) S. 94 - 101
Hauptverfasser: Dingle, Adam, Krulis, Martin
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: ACM 20.04.2024
Schlagworte:
Codes
coding
Computational modeling
Computer languages
Conferences
Education
Encoding
large language model
Large language models
LLM
Natural language processing
programming
Programming profession
student assignment
teaching
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating on coding tests or homework assignments. At present, knowledge about the exact capabilities and limitations of state-of-the-art LLMs is still inadequate. Furthermore, their capabilities have been changing quickly with each new release. In this paper, we present a dataset of 559 programming exercises in 10 programming languages collected from a system for evaluating coding assignments at our university. We have experimented with four well-known LLMs (GPT-3.5, GPT-4, Codey, Code Llama) and asked them to solve these assignments. The evaluation results are intriguing and provide insights into the strengths and weaknesses of the models. In particular, GPT-4 (which performed the best) is currently capable of solving 55% of all our exercises and achieved an average score of 86% on exercises from the introductory programming course (using the best of five generated solutions).CCS CONCEPTS* Computing methodologies → Natural language processing; * General and reference → Evaluation; * Applied computing→ Education.
DOI:10.1145/3643795.3648389