Tackling Students' Coding Assignments with LLMs

State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating o...

Full description

Saved in:
Bibliographic Details
Published in:2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) pp. 94 - 101
Main Authors: Dingle, Adam, Krulis, Martin
Format: Conference Proceeding
Language:English
Published: ACM 20.04.2024
Subjects:
Codes
coding
Computational modeling
Computer languages
Conferences
Education
Encoding
large language model
Large language models
LLM
Natural language processing
programming
Programming profession
student assignment
teaching
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating on coding tests or homework assignments. At present, knowledge about the exact capabilities and limitations of state-of-the-art LLMs is still inadequate. Furthermore, their capabilities have been changing quickly with each new release. In this paper, we present a dataset of 559 programming exercises in 10 programming languages collected from a system for evaluating coding assignments at our university. We have experimented with four well-known LLMs (GPT-3.5, GPT-4, Codey, Code Llama) and asked them to solve these assignments. The evaluation results are intriguing and provide insights into the strengths and weaknesses of the models. In particular, GPT-4 (which performed the best) is currently capable of solving 55% of all our exercises and achieved an average score of 86% on exercises from the introductory programming course (using the best of five generated solutions).CCS CONCEPTS* Computing methodologies → Natural language processing; * General and reference → Evaluation; * Applied computing→ Education.
DOI:10.1145/3643795.3648389