Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks

Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software deve...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Proceedings of the ACM on software engineering Ročník 1; číslo FSE; s. 699 - 721
Hlavní autori: Wang, Wei, Ning, Huilong, Zhang, Gaowei, Liu, Libo, Wang, Yi
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York, NY, USA ACM 12.07.2024
Predmet:
Human-centered computing
Laboratory experiments
Software and its engineering
Software development methods
Software development techniques
controlled experiment
large langauge models
human-AI collaboration
software development task
ISSN:2994-970X, 2994-970X
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software development and replace human software developers. Nevertheless, there are in a lack of serious investigation into the capability of these LLM techniques in fulfilling software development tasks. In a controlled 2 × 2 between-subject experiment with 109 participants, we examined whether and to what degree working with ChatGPT was helpful in the coding task and typical software development task and how people work with ChatGPT. We found that while ChatGPT performed well in solving simple coding problems, its performance in supporting typical software development tasks was not that good. We also observed the interactions between participants and ChatGPT and found the relations between the interactions and the outcomes. Our study thus provides first-hand insights into using ChatGPT to fulfill software engineering tasks with real-world developers and motivates the need for novel interaction mechanisms that help developers effectively work with large language models to achieve desired outcomes.
ISSN:2994-970X
2994-970X
DOI:10.1145/3643758