Automated Behavioral Testing of Refactoring Engines

Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and tra...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on software engineering Vol. 39; no. 2; pp. 147 - 162
Main Authors:	Soares, G., Gheyi, R., Massoni, T.
Format:	Journal Article
Language:	English
Published:	New York IEEE 01.02.2013 IEEE Computer Society
Subjects:	automated testing Automatic programming Automation Code reuse Computer bugs Computer programming Debugging Engines Java Metals program generation Refactoring Software engineering Software quality Studies Testing Unified modeling language
ISSN:	0098-5589, 1939-3520
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes.
AbstractList	Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes. Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes. [PUBLICATION ABSTRACT]
Author	Soares, G. Gheyi, R. Massoni, T.
Author_xml	– sequence: 1 givenname: G. surname: Soares fullname: Soares, G. email: gsoares@dsc.ufcg.edu.br organization: Dept. of Comput. & Syst. (DSC, Fed. Univ. of Campina Grande (UFCG), Campina Grande, Brazil – sequence: 2 givenname: R. surname: Gheyi fullname: Gheyi, R. email: rohit@dsc.ufcg.edu.br organization: Dept. of Comput. & Syst. (DSC, Fed. Univ. of Campina Grande (UFCG), Campina Grande, Brazil – sequence: 3 givenname: T. surname: Massoni fullname: Massoni, T. email: massoni@dsc.ufcg.edu.br organization: Dept. of Comput. & Syst. (DSC, Fed. Univ. of Campina Grande (UFCG), Campina Grande, Brazil
BookMark	eNp1kE1LAzEQhoNUsK2ePHpZ8ChbM8nmY4-11A8oCFrPIZudrSntpmZTwX_vlooHwdPwwjPzDs-IDNrQIiGXQCcAtLxdvs4njAKbQHlChlDyMueC0QEZUlrqXAhdnpFR160ppUIpMSR8uk9haxPW2R2-208fot1kS-ySb1dZaLIXbKxLIR7ivF35FrtzctrYTYcXP3NM3u7ny9ljvnh-eJpNF7njhUq5qJSTGgsqnXRFJYtaS9QMFdbALEfKBShdc1ZVrIaq0gx0UwsL2lbOMeBjcn28u4vhY9-_ZNZhH9u-0gCXkhZKgegpOFIuhq6L2Bjnk00-tClavzFAzcGN6d2YgxvTaxmTmz87u-i3Nn79Q18daY-Iv6QEJUoA_g1GTm6A
CODEN	IESEDJ
CitedBy_id	crossref_primary_10_1016_j_infsof_2016_04_016 crossref_primary_10_1109_TSE_2017_2688333 crossref_primary_10_1016_j_infsof_2020_106332 crossref_primary_10_1109_ACCESS_2021_3120414 crossref_primary_10_1145_3487062 crossref_primary_10_14201_ADCAIJ2018733146 crossref_primary_10_1109_TSE_2017_2693982 crossref_primary_10_1016_j_scico_2020_102494 crossref_primary_10_1016_j_infsof_2017_11_010 crossref_primary_10_1109_ACCESS_2019_2919203 crossref_primary_10_1016_j_jss_2016_02_001 crossref_primary_10_1016_j_jss_2017_11_073 crossref_primary_10_1007_s11390_019_1917_9 crossref_primary_10_1007_s10817_023_09692_0 crossref_primary_10_1109_TR_2020_2979815 crossref_primary_10_1145_3156016 crossref_primary_10_1155_2018_9251762 crossref_primary_10_1109_TSE_2019_2948351 crossref_primary_10_1002_stvr_1603 crossref_primary_10_3390_app122312217 crossref_primary_10_1109_ACCESS_2021_3140036 crossref_primary_10_1109_TSE_2014_2357438 crossref_primary_10_1016_j_eswa_2014_09_022 crossref_primary_10_1145_3011286_3011294 crossref_primary_10_1109_TSE_2015_2448531 crossref_primary_10_1002_smr_1951 crossref_primary_10_1145_3280985 crossref_primary_10_1007_s10664_023_10287_x crossref_primary_10_1016_j_scico_2013_11_001 crossref_primary_10_1109_TSE_2017_2679742 crossref_primary_10_1142_S0218194018500365 crossref_primary_10_1145_3170492_3136053 crossref_primary_10_1002_spe_2228 crossref_primary_10_1016_j_asej_2017_03_002 crossref_primary_10_1145_2932631 crossref_primary_10_1109_TR_2021_3061618 crossref_primary_10_1109_TSE_2019_2929761 crossref_primary_10_1016_j_infsof_2018_07_003
Cites_doi	10.1145/1449764.1449787 10.1145/1869459.1869485 10.1007/978-3-642-03013-0_17 10.1147/sj.94.0242 10.1145/267580.267590 10.1109/ASE.2001.989787 10.1109/TSE.2005.52 10.1109/ICSM.2011.6080784 10.1145/390016.808473 10.1109/MS.2010.63 10.1007/978-3-540-85373-2_2 10.1007/3-540-45832-8_22 10.1109/MS.2006.105 10.1007/978-3-642-14107-2_11 10.1145/1806799.1806835 10.1109/TSE.2004.1265817 10.1016/j.scico.2004.03.003 10.1145/1094811.1094832 10.1007/978-3-642-03013-0_19 10.1109/ICST.2010.64 10.1007/978-3-642-03013-0_18 10.1023/A:1008715808855 10.1145/337180.337616 10.1007/11531142_4 10.1145/271771.271803 10.1109/ICSM.2005.90 10.1007/978-3-642-00593-0_12 10.1109/TSE.1982.235428 10.1145/1481848.1481859 10.1109/ASE.2011.6100067 10.1145/949305.949308 10.1109/SEFM.2008.29 10.1109/ICSE.2007.37 10.1145/1810295.1810353 10.1145/1287624.1287651
ContentType	Journal Article
Copyright	Copyright IEEE Computer Society Feb 2013
Copyright_xml	– notice: Copyright IEEE Computer Society Feb 2013
DBID	97E RIA RIE AAYXX CITATION JQ2 K9.
DOI	10.1109/TSE.2012.19
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) Online IEEE Electronic Library (IEL) CrossRef ProQuest Computer Science Collection ProQuest Health & Medical Complete (Alumni)
DatabaseTitle	CrossRef ProQuest Health & Medical Complete (Alumni) ProQuest Computer Science Collection
DatabaseTitleList	ProQuest Health & Medical Complete (Alumni)
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Computer Science
EISSN	1939-3520
EndPage	162
ExternalDocumentID	2991960451 10_1109_TSE_2012_19 6175911
Genre	orig-research Feature
GroupedDBID	--Z -DZ -~X .4S .DC 0R~ 29I 3EH 4.4 5GY 5VS 6IK 7WY 7X7 85S 88E 88I 8FE 8FG 8FI 8FJ 8FL 8G5 8R4 8R5 97E 9M8 AAJGR AARMG AASAJ AAWTH ABAZT ABFSI ABJCF ABPPZ ABQJQ ABUWG ABVLG ACGFO ACGOD ACIWK ACNCT ADBBV AENEX AETIX AFKRA AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ARAPS ARCSS ASUFR ATWAV AZQEC BEFXN BENPR BEZIV BFFAM BGLVJ BGNUA BKEBE BKOMP BPEOZ BPHCQ BVXVI CCPQU CS3 DU5 DWQXO E.L EBS EDO EJD FRNLG FYUFA GNUQQ GROUPED_ABI_INFORM_RESEARCH GUQSH HCIFZ HMCUK HZ~ H~9 I-F IBMZZ ICLAB IEDLZ IFIPE IFJZH IPLJI ITG ITH JAVBF K60 K6V K6~ K7- L6V LAI M0C M1P M1Q M2O M2P M43 M7S MS~ O9- OCL OHT P2P P62 PHGZM PHGZT PJZUB PPXIY PQBIZ PQBZA PQGLB PQQKQ PROAC PSQYO PTHSS PUEGO Q2X RIA RIE RNI RNS RXW RZB S10 TAE TN5 TWZ UHB UKHRP UPT UQL VH1 WH7 XOL YYP YZZ ZCG AAYXX AFFHD CITATION JQ2 K9.
ID	FETCH-LOGICAL-c347t-5b7c68e406c6c4b64d86e82e7ed12a3e035178d32bb2d1bb8218fd5a18abcc213
IEDL.DBID	RIE
ISICitedReferencesCount	83
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000314174500002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	0098-5589
IngestDate	Sun Nov 30 04:20:24 EST 2025 Tue Nov 18 22:04:50 EST 2025 Sat Nov 29 08:05:25 EST 2025 Wed Aug 27 02:47:43 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	2
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c347t-5b7c68e406c6c4b64d86e82e7ed12a3e035178d32bb2d1bb8218fd5a18abcc213
Notes	SourceType-Scholarly Journals-1 ObjectType-Feature-1 content type line 14
PQID	1366047715
PQPubID	21418
PageCount	16
ParticipantIDs	ieee_primary_6175911 crossref_citationtrail_10_1109_TSE_2012_19 proquest_journals_1366047715 crossref_primary_10_1109_TSE_2012_19
PublicationCentury	2000
PublicationDate	2013-02-01
PublicationDateYYYYMMDD	2013-02-01
PublicationDate_xml	– month: 02 year: 2013 text: 2013-02-01 day: 01
PublicationDecade	2010
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on software engineering
PublicationTitleAbbrev	TSE
PublicationYear	2013
Publisher	IEEE IEEE Computer Society
Publisher_xml	– name: IEEE – name: IEEE Computer Society
References	ref13 ref12 ref15 Cornélio (ref9) 2004 ref14 ref11 ref10 ref17 ref16 ref18 Hoffman (ref35) 2011; 41 ref46 ref45 ref48 ref47 ref42 ref41 Jackson (ref19) 2006 ref44 Opdyke (ref1) 1992 ref43 ref49 ref8 ref3 Fowler (ref2) 1999 ref40 (ref5) 2011 Celentano (ref33) 1980; 10 Gosling (ref21) 2005 ref34 Schäfer (ref28) 2011 ref37 ref36 ref31 (ref24) 2011 ref30 ref32 Mckeeman (ref27) 1998; 10 ref39 ref38 (ref7) 2011 ref23 ref26 (ref4) 2011 ref25 ref20 ref22 (ref6) 2011 ref29
References_xml	– ident: ref12 doi: 10.1145/1449764.1449787 – ident: ref14 doi: 10.1145/1869459.1869485 – ident: ref13 doi: 10.1007/978-3-642-03013-0_17 – ident: ref46 doi: 10.1147/sj.94.0242 – ident: ref31 doi: 10.1145/267580.267590 – ident: ref48 doi: 10.1109/ASE.2001.989787 – ident: ref49 doi: 10.1109/TSE.2005.52 – year: 2011 ident: ref7 article-title: JBuilder – volume-title: Refactoring: Improving the Design of Existing Code year: 1999 ident: ref2 – volume-title: Software Abstractions: Logic, Language, and Analysis year: 2006 ident: ref19 – year: 2011 ident: ref6 article-title: IntelliJ Idea – year: 2011 ident: ref5 article-title: NetBeans IDE – ident: ref26 doi: 10.1109/ICSM.2011.6080784 – ident: ref30 doi: 10.1145/390016.808473 – ident: ref18 doi: 10.1109/MS.2010.63 – volume: 10 start-page: 897 issue: 11 year: 1980 ident: ref33 article-title: Compiler Testing Using a Sentence Generator publication-title: Software: Practice and Experience – ident: ref44 doi: 10.1007/978-3-540-85373-2_2 – ident: ref39 doi: 10.1007/3-540-45832-8_22 – ident: ref25 doi: 10.1109/MS.2006.105 – ident: ref40 doi: 10.1007/978-3-642-14107-2_11 – ident: ref32 doi: 10.1145/1806799.1806835 – ident: ref3 doi: 10.1109/TSE.2004.1265817 – ident: ref8 doi: 10.1016/j.scico.2004.03.003 – year: 2011 ident: ref24 article-title: JDT Core Component – year: 2004 ident: ref9 article-title: Refactorings as Formal Refinements – ident: ref37 doi: 10.1145/1094811.1094832 – volume: 10 start-page: 100 issue: 1 year: 1998 ident: ref27 article-title: Differential Testing for Software publication-title: Digital Technical J. – ident: ref15 doi: 10.1007/978-3-642-03013-0_19 – volume-title: The Java Language Specification year: 2005 ident: ref21 – volume: 41 start-page: 427 year: 2011 ident: ref35 article-title: Grammar-Based Test Generation with YouGen publication-title: Software: Practice and Experience – ident: ref47 doi: 10.1109/ICST.2010.64 – year: 2011 ident: ref28 article-title: Refactoring Bugs – year: 2011 ident: ref4 article-title: Eclipse Project – ident: ref43 doi: 10.1007/978-3-642-03013-0_18 – ident: ref38 doi: 10.1023/A:1008715808855 – ident: ref20 doi: 10.1145/337180.337616 – ident: ref36 doi: 10.1007/11531142_4 – ident: ref45 doi: 10.1145/271771.271803 – ident: ref42 doi: 10.1109/ICSM.2005.90 – ident: ref23 doi: 10.1007/978-3-642-00593-0_12 – ident: ref34 doi: 10.1109/TSE.1982.235428 – ident: ref16 doi: 10.1145/1481848.1481859 – ident: ref41 doi: 10.1109/ASE.2011.6100067 – year: 1992 ident: ref1 article-title: Refactoring Object-Oriented Frameworks – ident: ref10 doi: 10.1145/949305.949308 – ident: ref11 doi: 10.1109/SEFM.2008.29 – ident: ref22 doi: 10.1109/ICSE.2007.37 – ident: ref29 doi: 10.1145/1810295.1810353 – ident: ref17 doi: 10.1145/1287624.1287651
SSID	ssj0005775
Score	2.3732495
Snippet	Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	147
SubjectTerms	automated testing Automatic programming Automation Code reuse Computer bugs Computer programming Debugging Engines Java Metals program generation Refactoring Software engineering Software quality Studies Testing Unified modeling language
Title	Automated Behavioral Testing of Refactoring Engines
URI	https://ieeexplore.ieee.org/document/6175911 https://www.proquest.com/docview/1366047715
Volume	39
WOSCitedRecordID	wos000314174500002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1939-3520 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0005775 issn: 0098-5589 databaseCode: RIE dateStart: 19750101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8MwGA5zePDi1ClOp_Swk9itSfN5nLLhQYbolN1K81ERpJWt8_ebpB8KevHWwlsIb5o8b5v3eR4ARlCnOlKShJJkJsQqhaEF4TQUxBbPOkOKeXr0yz1bLPhqJR464LrlwhhjfPOZGbtLf5avC7V1v8omFm2JcETeHcZoxdX6budgjDT6mIRwUXPxYCQmy6eZ6-FCYyen8wN9vJ3Krz3YA8u8978hHYD9uoAMptWMH4KOyY9ArzFnCOq12gfxdFsWthw1OrhpqfjB0olq5K9BkQWPpvLacbeVKuHmGDzPZ8vbu7B2SAhVjFkZEskU5caCsqIKS4o1p4Yjw4yGKI2NOyZkXMdISqShlNwCeqZJCnkqlUIwPgHdvMjNKQgkF0pb9MZCekkZIbBMJWEmoxbdMjwAV03mElXLhzsXi_fEf0ZEIrFpTlyaEygGYNQGf1SqGX-H9V1G25A6mQMwbKYkqVfUJoExpRFmDJKzv586B3vIW1W4VpMh6JbrrbkAu-qzfNusL_3L8gXckL0U
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1dS8MwFL3IFPTFqVOcn33Yk9itSZMmeZwyUZxDdMreSvNREWSTrfP3m6TdFNyLby3cQrhpcm6be84BaCGd6UhJGkqam5CoDIUWhLNQUFs86xwr5unRr302GPDRSDyuweWSC2OM8c1npu0u_Vm-nqi5-1XWsWhLhSPyrlNCcFSytX4aOhijC4VMSrmo2HgoEp3hc891ceG2E9T5hT_eUOXPLuyh5ab-v0HtwHZVQgbdcs53Yc2M96C-sGcIqtXagLg7Lya2IDU6uFqS8YOhk9UYvwWTPHgypduOuy11CWf78HLTG17fhpVHQqhiwoqQSqYSbiwsq0QRmRDNE8OxYUYjnMXGHRQyrmMsJdZISm4hPdc0QzyTSmEUH0BtPBmbQwgkF0pb_CZCelEZIYjMJGUmTyy-5aQJF4vMpaoSEHc-Fh-p_5CIRGrTnLo0p0g0obUM_ix1M1aHNVxGlyFVMptwspiStFpTsxTFSRIRxhA9Wv3UOWzeDh_6af9ucH8MW9gbV7jGkxOoFdO5OYUN9VW8z6Zn_sX5Bt-VwFs
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automated+Behavioral+Testing+of+Refactoring+Engines&rft.jtitle=IEEE+transactions+on+software+engineering&rft.au=Soares%2C+Gustavo&rft.au=Gheyi%2C+Rohit&rft.au=Massoni%2C+Tiago&rft.date=2013-02-01&rft.issn=0098-5589&rft.volume=39&rft.issue=2&rft.spage=147&rft.epage=162&rft_id=info:doi/10.1109%2FTSE.2012.19&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TSE_2012_19
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0098-5589&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0098-5589&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0098-5589&client=summon