Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks

Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software deve...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the ACM on software engineering Vol. 1; no. FSE; pp. 699 - 721
Main Authors: Wang, Wei, Ning, Huilong, Zhang, Gaowei, Liu, Libo, Wang, Yi
Format: Journal Article
Language:English
Published: New York, NY, USA ACM 12.07.2024
Subjects:
Human-centered computing
Laboratory experiments
Software and its engineering
Software development methods
Software development techniques
controlled experiment
large langauge models
human-AI collaboration
software development task
ISSN:2994-970X, 2994-970X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software development and replace human software developers. Nevertheless, there are in a lack of serious investigation into the capability of these LLM techniques in fulfilling software development tasks. In a controlled 2 × 2 between-subject experiment with 109 participants, we examined whether and to what degree working with ChatGPT was helpful in the coding task and typical software development task and how people work with ChatGPT. We found that while ChatGPT performed well in solving simple coding problems, its performance in supporting typical software development tasks was not that good. We also observed the interactions between participants and ChatGPT and found the relations between the interactions and the outcomes. Our study thus provides first-hand insights into using ChatGPT to fulfill software engineering tasks with real-world developers and motivates the need for novel interaction mechanisms that help developers effectively work with large language models to achieve desired outcomes.
ISSN:2994-970X
2994-970X
DOI:10.1145/3643758