Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study

Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive perf...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings / International Conference on Software Engineering pp. 390 - 402
Main Authors: Guo, Qi, Cao, Junming, Xie, Xiaofei, Liu, Shangqing, Li, Xiaohong, Chen, Bihuan, Peng, Xin
Format: Conference Proceeding
Language:English
Published: ACM 14.04.2024
Subjects:
Activities Automated Code Refinement
Automating Code Review
Benchmark testing
Chatbots
ChatGPT Empirical Study
Codes
Focusing
Reviews
Software
Task analysis
ISSN:1558-1225
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeRe-viewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT's underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions.
AbstractList Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeRe-viewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT's underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions.
Author Li, Xiaohong
Liu, Shangqing
Chen, Bihuan
Peng, Xin
Guo, Qi
Cao, Junming
Xie, Xiaofei
Author_xml – sequence: 1
  givenname: Qi
  surname: Guo
  fullname: Guo, Qi
  organization: Tianjin University,Tianjin,China
– sequence: 2
  givenname: Junming
  surname: Cao
  fullname: Cao, Junming
  organization: Fudan University,Shanghai,China
– sequence: 3
  givenname: Xiaofei
  surname: Xie
  fullname: Xie, Xiaofei
  organization: Singapore Management University,Singapore
– sequence: 4
  givenname: Shangqing
  surname: Liu
  fullname: Liu, Shangqing
  organization: Nanyang Technological University,Singapore
– sequence: 5
  givenname: Xiaohong
  surname: Li
  fullname: Li, Xiaohong
  organization: Tianjin University,Tianjin,China
– sequence: 6
  givenname: Bihuan
  surname: Chen
  fullname: Chen, Bihuan
  organization: Fudan University,Shanghai,China
– sequence: 7
  givenname: Xin
  surname: Peng
  fullname: Peng, Xin
  organization: Fudan University,Shanghai,China
BookMark eNotj0FLAzEUhKMoWGvPXjzkD2xN8pJN4q0saxUKFlvP5bV5awPdbNmmYP-9C3qaw8w3zNyzm9QlYuxRiqmU2jyD8dYImEKpAER5xSbeeqeFsEJJq6_ZSBrjCqmUuWOT0yluhdFgbKlhxFb1z_HQ9TF987wnvuwypRzxwLuGV3vM8-Wax8Rn59y1mCnwqgvEP6mJidoh-sJnidftMfZxN1CrfA6XB3bb4OFEk38ds6_Xel29FYuP-Xs1WxSoQJWFUhiU8GYXlFboSwseAS2gwmFaCOSbxnm5tSRJe9ClaKyw3qJD5wYDxuzprzcS0ebYxxb7y0YO75yUHn4B8xxQdw
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1145/3597503.3623306
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798400702174
EISSN 1558-1225
EndPage 402
ExternalDocumentID 10548119
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  grantid: 61872262
  funderid: 10.13039/501100001809
– fundername: National Key R&D Project
  grantid: 2021YFF1201102
  funderid: 10.13039/501100012166
– fundername: National Research Foundation
  funderid: 10.13039/100011512
GroupedDBID -~X
.4S
.DC
29O
5VS
6IE
6IF
6IH
6IK
6IL
6IM
6IN
8US
AAJGR
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
ARCSS
AVWKF
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
EDO
FEDTE
I-F
IEGSK
IJVOP
IPLJI
M43
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-a2326-22ad2095cd242a96739a3a73a2a643dde9ff891b7e1e493460f70797a8a888913
IEDL.DBID RIE
IngestDate Wed Aug 27 01:53:12 EDT 2025
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a2326-22ad2095cd242a96739a3a73a2a643dde9ff891b7e1e493460f70797a8a888913
OpenAccessLink https://ink.library.smu.edu.sg/sis_research/9175
PageCount 13
ParticipantIDs ieee_primary_10548119
PublicationCentury 2000
PublicationDate 2024-April-14
PublicationDateYYYYMMDD 2024-04-14
PublicationDate_xml – month: 04
  year: 2024
  text: 2024-April-14
  day: 14
PublicationDecade 2020
PublicationTitle Proceedings / International Conference on Software Engineering
PublicationTitleAbbrev ICSE
PublicationYear 2024
Publisher ACM
Publisher_xml – name: ACM
SSID ssib054357643
ssib055306466
ssj0006499
Score 2.5492156
Snippet Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone...
SourceID ieee
SourceType Publisher
StartPage 390
SubjectTerms Activities Automated Code Refinement
Automating Code Review
Benchmark testing
Chatbots
ChatGPT Empirical Study
Codes
Focusing
Reviews
Software
Task analysis
Title Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
URI https://ieeexplore.ieee.org/document/10548119
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ07T8MwEMctqBiYyqOItzywpsR2YtdsVdXCVEVQpG6VE59FJUiqkiLx7Tm7aYGBgS2KEiXy4-5n--5_hNzk3OHI4SpySex3q3IRaZUWkUoBQBtm8cVQbEKNx73pVGdNsnrIhcFHQvAZdP1lOMu3VbHyW2U4w5GvmRf53FVKrpO1NoMnRb-vfmhL-XI4MvGs0phliWzfaPuwJL0VSNJpLLpowHFJL38VVwm-ZdT-518dkM53lh7Ntv7nkOxAeUTamzINtJm1x-RpG2dHkfZoVtU-Qsi80srRwYup77MJnZe0v6orxFewdFBZoI_gEED9p-9ov6TDt8U8qIlQH3n42SHPo-Fk8BA1tRQig8wkI86N5YhThUWfbLRUQhthlDDcYKOhjdPO9TTLFTBItEhk7Lx2njI9g2tkzcQJaZVVCaeE2iK3rLAih1wlxoGRWqTAdax5WiDRnZGOb6TZYi2XMdu0z_kf9y_IPkdS8Ec0LLkkrXq5giuyV3zU8_fldejkL2T0pKU
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA8yBT3Nj4nf5uC1s_lqFm9jbE6co-iE3UbaJDjQdsxO8L_3peumHjx4K6Wl5SV57_eS934_hK4S6mDmUBk4HvrdqoQFSoo0kMJaqzQx8GIpNiGHw9Z4rOKqWb3shYFHyuIz2_SX5Vm-ydOF3yqDFQ74mniSz03BIfFZtmutpo-AyC9_sEt5QZyIe7RSOeYI0H3F7kO4uGaApUXImuDCIamPfsmrlNGlV__nf-2ixnefHo7XEWgPbdhsH9VXQg24WrcH6GldaYcB7-E4L3yNkH7FucOdF13cxiM8zXB7UeQAYK3BndxY_GgdQFD_6RvcznD3bTYt-USwrz38bKDnXnfU6QeVmkKgATVFAaXaUABUqYGorFUkmdJMS6apBqOBl1POtRRJpCWWK8aj0Hn2PKlbGrJkRdghqmV5Zo8QNmliSGpYYhPJtbM6UkxYqkJFRQqY7hg1vJEmsyVhxmRln5M_7l-i7f7oYTAZ3A3vT9EOBdzgD2wIP0O1Yr6w52gr_Sim7_OLcsC_AChMp-w
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=Exploring+the+Potential+of+ChatGPT+in+Automated+Code+Refinement%3A+An+Empirical+Study&rft.au=Guo%2C+Qi&rft.au=Cao%2C+Junming&rft.au=Xie%2C+Xiaofei&rft.au=Liu%2C+Shangqing&rft.date=2024-04-14&rft.pub=ACM&rft.eissn=1558-1225&rft.spage=390&rft.epage=402&rft_id=info:doi/10.1145%2F3597503.3623306&rft.externalDocID=10548119