PyDex: Repairing Bugs in Introductory Python Assignments using LLMs

Students often make mistakes in their introductory programming assignments as part of their learning process. Unfortunately, providing custom repairs for these mistakes can require a substantial amount of time and effort from class instructors. Automated program repair (APR) techniques can be used t...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Proceedings of ACM on programming languages Ročník 8; číslo OOPSLA1; s. 1100 - 1124
Hlavní autoři: Zhang, Jialu, Cambronero, José Pablo, Gulwani, Sumit, Le, Vu, Piskac, Ruzica, Soares, Gustavo, Verbruggen, Gust
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York, NY, USA ACM 29.04.2024
Témata:
Artificial intelligence
Computer science education
Computing methodologies
Social and professional topics
automated program repair
AI for programming education
large language models
ISSN:2475-1421, 2475-1421
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Students often make mistakes in their introductory programming assignments as part of their learning process. Unfortunately, providing custom repairs for these mistakes can require a substantial amount of time and effort from class instructors. Automated program repair (APR) techniques can be used to synthesize such fixes. Prior work has explored the use of symbolic and neural techniques for APR in the education domain. Both types of approaches require either substantial engineering efforts or large amounts of data and training. We propose to use a large language model trained on code, such as Codex (a version of GPT), to build an APR system – PyDex – for introductory Python programming assignments. Our system can fix both syntactic and semantic mistakes by combining multi-modal prompts, iterative querying, test-case-based selection of few-shots, and program chunking. We evaluate PyDex on 286 real student programs and compare to three baselines, including one that combines a state-of-the-art Python syntax repair engine, BIFI, and a state-of-the-art Python semantic repair engine for student assignments, Refactory. We find that PyDex can fix more programs and produce smaller patches on average.
ISSN:2475-1421
2475-1421
DOI:10.1145/3649850