Tackling Students' Coding Assignments with LLMs - Dingle, Adam; Krulis, Martin (2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), 2024)
State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating on coding tests or homework assignments. At present, knowledge about the exact capabilities and limitations of state-of-the-art LLMs is still inadequate. Furthermore, their capabilities have been changing quickly with each new release. In this paper, we present a dataset of 559 programming exercises in 10 programming languages collected from a system for evaluating coding assignments at our university. We have experimented with four well-known LLMs (GPT-3.5, GPT-4, Codey, Code Llama) and asked them to solve these assignments. The evaluation results are intriguing and provide insights into the strengths and weaknesses of the models. In particular, GPT-4 (which performed the best) is currently capable of solving 55% of all our exercises and achieved an average score of 86% on exercises from the introductory programming course (using the best of five generated solutions).CCS CONCEPTS* Computing methodologies → Natural language processing; * General and reference → Evaluation; * Applied computing→ Education.