Tackling Students' Coding Assignments with LLMs

State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating o...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) s. 94 - 101
Hlavní autoři: Dingle, Adam, Krulis, Martin
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: ACM 20.04.2024
Témata:
Codes
coding
Computational modeling
Computer languages
Conferences
Education
Encoding
large language model
Large language models
LLM
Natural language processing
programming
Programming profession
student assignment
teaching
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating on coding tests or homework assignments. At present, knowledge about the exact capabilities and limitations of state-of-the-art LLMs is still inadequate. Furthermore, their capabilities have been changing quickly with each new release. In this paper, we present a dataset of 559 programming exercises in 10 programming languages collected from a system for evaluating coding assignments at our university. We have experimented with four well-known LLMs (GPT-3.5, GPT-4, Codey, Code Llama) and asked them to solve these assignments. The evaluation results are intriguing and provide insights into the strengths and weaknesses of the models. In particular, GPT-4 (which performed the best) is currently capable of solving 55% of all our exercises and achieved an average score of 86% on exercises from the introductory programming course (using the best of five generated solutions).CCS CONCEPTS* Computing methodologies → Natural language processing; * General and reference → Evaluation; * Applied computing→ Education.
DOI:10.1145/3643795.3648389