On the Efficiency of Test Suite based Program Repair A Systematic Assessment of 16 Automated Repair Systems for Java Programs

Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been proposed, which leverage test suites as a weak, but affordable, approximation to program specifications. Although the literature regularly sets new r...

Full description

Saved in:
Bibliographic Details
Published in:2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) pp. 615 - 627
Main Authors: Liu, Kui, Wang, Shangwen, Kisub Kim, Anil Koyuncu, Bissyande, Tegawende F., Kim, Dongsun, Wu, Peng, Klein, Jacques, Mao, Xiaoguang, Traon, Yves Le
Format: Conference Proceeding
Language:English
Published: ACM 01.10.2020
Subjects:
Efficiency
Empirical assessment
Patch generation
Program repair
ISSN:1558-1225
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been proposed, which leverage test suites as a weak, but affordable, approximation to program specifications. Although the literature regularly sets new records on the number of benchmark bugs that can be fixed, several studies increasingly raise concerns about the limitations and biases of state-of-the-art approaches. For example, the correctness of generated patches has been questioned in a number of studies, while other researchers pointed out that evaluation schemes may be misleading with respect to the processing of fault localization results. Nevertheless, there is little work addressing the efficiency of patch generation, with regard to the practicality of program repair. In this paper, we fill this gap in the literature, by providing an extensive review on the efficiency of test suite based program repair. Our objective is to assess the number of generated patch candidates, since this information is correlated to (1) the strategy to traverse the search space efficiently in order to select sensical repair attempts, (2) the strategy to minimize the test effort for identifying a plausible patch, (3) as well as the strategy to prioritize the generation of a correct patch. To that end, we performa large-scale empirical study on the efficiency, in terms of quantity of generated patch candidates of the 16 open-source repair tools for Java programs. The experiments are carefully conducted under the same fault localization configurations to limit biases. Eventually, among other findings, we note that: (1) many irrelevant patch candidates are generated by changing wrong code locations; (2) however, if the search space is carefully triaged, fault localization noise has little impact on patch generation efficiency; (3) yet, current template-based repair systems, which are known to be most effective in fixing a large number of bugs, are actually least efficient as they tend to generate majoritarily irrelevant patch candidates.
AbstractList Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been proposed, which leverage test suites as a weak, but affordable, approximation to program specifications. Although the literature regularly sets new records on the number of benchmark bugs that can be fixed, several studies increasingly raise concerns about the limitations and biases of state-of-the-art approaches. For example, the correctness of generated patches has been questioned in a number of studies, while other researchers pointed out that evaluation schemes may be misleading with respect to the processing of fault localization results. Nevertheless, there is little work addressing the efficiency of patch generation, with regard to the practicality of program repair. In this paper, we fill this gap in the literature, by providing an extensive review on the efficiency of test suite based program repair. Our objective is to assess the number of generated patch candidates, since this information is correlated to (1) the strategy to traverse the search space efficiently in order to select sensical repair attempts, (2) the strategy to minimize the test effort for identifying a plausible patch, (3) as well as the strategy to prioritize the generation of a correct patch. To that end, we performa large-scale empirical study on the efficiency, in terms of quantity of generated patch candidates of the 16 open-source repair tools for Java programs. The experiments are carefully conducted under the same fault localization configurations to limit biases. Eventually, among other findings, we note that: (1) many irrelevant patch candidates are generated by changing wrong code locations; (2) however, if the search space is carefully triaged, fault localization noise has little impact on patch generation efficiency; (3) yet, current template-based repair systems, which are known to be most effective in fixing a large number of bugs, are actually least efficient as they tend to generate majoritarily irrelevant patch candidates.
Author Traon, Yves Le
Kim, Dongsun
Klein, Jacques
Mao, Xiaoguang
Liu, Kui
Wang, Shangwen
Kisub Kim, Anil Koyuncu
Wu, Peng
Bissyande, Tegawende F.
Author_xml – sequence: 1
  givenname: Kui
  surname: Liu
  fullname: Liu, Kui
  email: brucekuiliu@gmail.com
  organization: Nanjing University of Aeronautics and Astronautics,China
– sequence: 2
  givenname: Shangwen
  surname: Wang
  fullname: Wang, Shangwen
  email: wangshangwen13@nudt.edu.cn
  organization: National University of Defense Technology,China
– sequence: 3
  givenname: Anil Koyuncu
  surname: Kisub Kim
  fullname: Kisub Kim, Anil Koyuncu
  email: anil.koyuncu@uni.lu
  organization: University of Luxembourg,Luxembourg
– sequence: 4
  givenname: Tegawende F.
  surname: Bissyande
  fullname: Bissyande, Tegawende F.
  email: tegawende.bissyande@uni.lu
  organization: University of Luxembourg,Luxembourg
– sequence: 5
  givenname: Dongsun
  surname: Kim
  fullname: Kim, Dongsun
  email: darkrsw@furiosa.ai
  organization: Furiosa.ai,Republic of Korea
– sequence: 6
  givenname: Peng
  surname: Wu
  fullname: Wu, Peng
  email: wupeng15@nudt.edu.cn
  organization: National University of Defense Technology,China
– sequence: 7
  givenname: Jacques
  surname: Klein
  fullname: Klein, Jacques
  email: jacques.klein@uni.lu
  organization: University of Luxembourg,Luxembourg
– sequence: 8
  givenname: Xiaoguang
  surname: Mao
  fullname: Mao, Xiaoguang
  email: xgmao@nudt.edu.cn
  organization: National University of Defense Technology,China
– sequence: 9
  givenname: Yves Le
  surname: Traon
  fullname: Traon, Yves Le
  email: yves.letraon@uni.lu
  organization: University of Luxembourg,Luxembourg
BookMark eNo1jEtLw0AUhUdR0FbXLtzcP5A6z0xmGUp9UajYui436R0dMUnJTIUu_O9GWheHA-fxjdhZ27XE2I3gEyG0uVPK2kKIiVIFH3TCRkPKlRVSqFN2KYwpMiGluWCjGD8557l27pL9LFpIHwQz70MdqK330HlYUUyw3IVEUGGkDbz03XuPDbzSFkMPJSz3MVGDKdRQxkgxNtSmv6vIodylbqiG23F-GEfwXQ_P-I3_uHjFzj1-Rbo--pi93c9W08dsvnh4mpbzDKW2KUMnlPAmrwkVoSPLPVaCcGM05dJUFZcoKzsEWhWoDGmnueS5K_LKcu3VmN0euIGI1ts-NNjv104WyimhfgGqhF83
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1145/3377811.3380338
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 1450371213
9781450371216
EISSN 1558-1225
EndPage 627
ExternalDocumentID 9283931
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  grantid: 61672529
  funderid: 10.13039/501100001809
– fundername: FNR
  grantid: 15/IS/10449467
  funderid: 10.13039/100007869
GroupedDBID -~X
.4S
.DC
123
23M
29O
5VS
6IE
6IF
6IH
6IK
6IL
6IM
6IN
8US
AAJGR
AAWTH
ABLEC
ADZIZ
AFFNX
ALMA_UNASSIGNED_HOLDINGS
APO
ARCSS
AVWKF
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
EDO
FEDTE
I-F
I07
IEGSK
IJVOP
IPLJI
M43
OCL
RIE
RIL
RIO
RNS
XOL
ID FETCH-LOGICAL-a247t-a9131f56cea3ea9e70fab1ead54e625bb02a2b7ead438a35e4940206986b704f3
IEDL.DBID RIE
ISICitedReferencesCount 98
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000652529800051&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:32:58 EDT 2025
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a247t-a9131f56cea3ea9e70fab1ead54e625bb02a2b7ead438a35e4940206986b704f3
PageCount 13
ParticipantIDs ieee_primary_9283931
PublicationCentury 2000
PublicationDate 2020-Oct.
PublicationDateYYYYMMDD 2020-10-01
PublicationDate_xml – month: 10
  year: 2020
  text: 2020-Oct.
PublicationDecade 2020
PublicationTitle 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)
PublicationTitleAbbrev ICSE
PublicationYear 2020
Publisher ACM
Publisher_xml – name: ACM
SSID ssj0006499
ssj0002870079
Score 2.5255253
Snippet Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been...
SourceID ieee
SourceType Publisher
StartPage 615
SubjectTerms Efficiency
Empirical assessment
Patch generation
Program repair
Title On the Efficiency of Test Suite based Program Repair A Systematic Assessment of 16 Automated Repair Systems for Java Programs
URI https://ieeexplore.ieee.org/document/9283931
WOSCitedRecordID wos000652529800051&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELVKxcBUoEV86wZG0tqxHcdjhVohhlKpRepW2YkjdUlQm3Tjv3NO08DAwhY58Sm6OL473713hDzpRKchrtuAc5YGQlMZWBvzgFrDZBJ6hvW4bjahZrN4tdLzDnlusTDOubr4zA39ZZ3LT4uk8kdlI422UHvQ9IlS0QGr1Z6n-IQd9RmnZheO0JVvqHyYkCPOlcdUDjEio9yDUX71UqlNybT3v5c4J4MfTB7MW2tzQTouvyS9Y1MGaP7RPvl6zwGdOpjU3BAeWAlFBkvc_GFRoX8J3m6lXpSvywJ0wM1mC2NYtJTOMG7ZOv1UFsG4Kgu8hdOaxxuec0CPF97M3hzF7QbkYzpZvrwGTY-FwIRClYHRjLNMRokz3BntFM2MZbi8pHAYGllLQxNahQOCx4ZLJ7SPOCMdR1ZRkfEr0s2L3F0TkJmWjsqECeME0yZGwagoZU2IQmJzQ_pem-vPA43GulHk7d_Dd-Qs9KFtXTd3T7rltnIP5DTZl5vd9rH-9t98RK7K
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELaqggRTgRbxxgMjaf1M4rFCrQqUUqlF6lbZiSN1SVCbdOO_c07TwMDCFjn2KTpdfHf2fd8h9KAiFTOwW49zGntCEekZE3KPGE1lxBzDelg2mwgmk3CxUNMGeqyxMNbasvjMdt1jeZcfZ1Hhjsp6CnyhcqDpAykEIzu0Vn2i4q7siLtzqvZhH4L5isyHCtnjPHCoyi7kZIQ7OMqvbiqlMxm2_vcZJ6jzg8rD09rfnKKGTc9Qa9-WAVd_aRt9vacYwjo8KNkhHLQSZwmew_aPZwVEmNh5rtiJcpVZGEJwvVrjPp7VpM64X_N1uqXUx_0iz-AVLKumV0znGGJe_KK3ei9u00Efw8H8aeRVXRY8zUSQe1pRThPpR1Zzq5UNSKINBQOTwkJyZAxhmpkABgQPNZdWKJdz-ir0TUBEws9RM81Se4GwTJS0REZUaCuo0iEIBkUFRjMQEupL1HbaXH7uiDSWlSKv_h6-R0ej-dt4OX6evF6jY-YS3bKK7gY183Vhb9FhtM1Xm_VdaQffv-uyEQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=2020+IEEE%2FACM+42nd+International+Conference+on+Software+Engineering+%28ICSE%29&rft.atitle=On+the+Efficiency+of+Test+Suite+based+Program+Repair+A+Systematic+Assessment+of+16+Automated+Repair+Systems+for+Java+Programs&rft.au=Liu%2C+Kui&rft.au=Wang%2C+Shangwen&rft.au=Kisub+Kim%2C+Anil+Koyuncu&rft.au=Bissyande%2C+Tegawende+F.&rft.date=2020-10-01&rft.pub=ACM&rft.eissn=1558-1225&rft.spage=615&rft.epage=627&rft_id=info:doi/10.1145%2F3377811.3380338&rft.externalDocID=9283931