Evaluating the Accuracy of Fault Localization Techniques

We investigate claims and assumptions made in several recent papers about fault localization (FL) techniques. Most of these claims have to do with evaluating FL accuracy. Our investigation centers on a new subject program having properties useful for FL experiments. We find that Tarantula (Jones et...

Full description

Saved in:
Bibliographic Details
Published in:2009 IEEE/ACM International Conference on Automated Software Engineering pp. 76 - 87
Main Authors: Ali, S., Andrews, J.H., Dhandapani, T., Wantao Wang
Format: Conference Proceeding
Language:English
Published: IEEE 01.11.2009
Subjects:
Computer science
Costs
Data analysis
Data mining
Fault localization
Genetic mutations
Humans
mutation analysis
Programming profession
Software engineering
Software testing
ISBN:1424452597, 9781424452590
ISSN:1938-4300
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract We investigate claims and assumptions made in several recent papers about fault localization (FL) techniques. Most of these claims have to do with evaluating FL accuracy. Our investigation centers on a new subject program having properties useful for FL experiments. We find that Tarantula (Jones et al.) works well on the program, and we show weak support for the assertion that coverage-based test suites help Tarantula to localize faults. Baudry et al. used automatically-generated mutants to evaluate the accuracy of an FL technique that generates many distinct scores for program locations. We find no evidence to suggest that the use of mutants for this purpose is invalid. However, we find evidence that the standard method for evaluating FL accuracy is unfairly biased toward techniques that generate many distinct scores, and we propose a fairer method of accuracy evaluation. Finally, Denmat et al. suggest that data mining techniques may apply to FL. We investigate this suggestion with the data mining tool Weka, using standard techniques for evaluating the accuracy of data mining classifiers. We find that standard classifiers suffer from the class imbalance problem. However, we find that adding cost information improves accuracy.
AbstractList We investigate claims and assumptions made in several recent papers about fault localization (FL) techniques. Most of these claims have to do with evaluating FL accuracy. Our investigation centers on a new subject program having properties useful for FL experiments. We find that Tarantula (Jones et al.) works well on the program, and we show weak support for the assertion that coverage-based test suites help Tarantula to localize faults. Baudry et al. used automatically-generated mutants to evaluate the accuracy of an FL technique that generates many distinct scores for program locations. We find no evidence to suggest that the use of mutants for this purpose is invalid. However, we find evidence that the standard method for evaluating FL accuracy is unfairly biased toward techniques that generate many distinct scores, and we propose a fairer method of accuracy evaluation. Finally, Denmat et al. suggest that data mining techniques may apply to FL. We investigate this suggestion with the data mining tool Weka, using standard techniques for evaluating the accuracy of data mining classifiers. We find that standard classifiers suffer from the class imbalance problem. However, we find that adding cost information improves accuracy.
Author Dhandapani, T.
Ali, S.
Andrews, J.H.
Wantao Wang
Author_xml – sequence: 1
  givenname: S.
  surname: Ali
  fullname: Ali, S.
  organization: Dept. of Comput. Sci., Univ. of Western Ontario, London, ON, Canada
– sequence: 2
  givenname: J.H.
  surname: Andrews
  fullname: Andrews, J.H.
  organization: Dept. of Comput. Sci., Univ. of Western Ontario, London, ON, Canada
– sequence: 3
  givenname: T.
  surname: Dhandapani
  fullname: Dhandapani, T.
  organization: Dept. of Comput. Sci., Univ. of Western Ontario, London, ON, Canada
– sequence: 4
  surname: Wantao Wang
  fullname: Wantao Wang
  organization: Dept. of Comput. Sci., Univ. of Western Ontario, London, ON, Canada
BookMark eNotj0tLw0AUhUesYFuzculm_kDinVdm7jKUVIWAC-u6TOZhIzHRPIT66w3o2ZzFd_jgbMiq67tAyC2DjDHA--KlzDgAZgYvSILagM5RCYNMXpINk1xKxRXqFVkzFCaVAuCaJOP4DksWhoatiSm_bTvbqene6HQKtHBuHqw70z7SvZ3biVa9s23zs0z6jh6CO3XN1xzGG3IVbTuG5L-35HVfHnaPafX88LQrqtQKUFPq61xC7r1ElUPNXfQaQEQuuJYKjMZgLRMBYp1rrx1THnnNc8sMc1GhEVty9-dtQgjHz6H5sMP5qKRgy2PxC2wMSTM
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ASE.2009.89
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9780769538914
0769538916
EndPage 87
ExternalDocumentID 5431780
Genre orig-research
GroupedDBID 29I
6IE
6IF
6IH
6IK
6IL
6IM
6IN
6J9
AAJGR
AAWTH
ABLEC
ACREN
ADYOE
ADZIZ
AFYQB
ALMA_UNASSIGNED_HOLDINGS
AMTXH
APO
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
M43
OCL
RIE
RIL
ID FETCH-LOGICAL-a305t-db6406dd49560b2cfd7003f2327450879eaa13e0fb67d7c15d92b26a181cf5983
IEDL.DBID RIE
ISBN 1424452597
9781424452590
ISICitedReferencesCount 34
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000278137400007&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 1938-4300
IngestDate Wed Aug 27 08:36:35 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a305t-db6406dd49560b2cfd7003f2327450879eaa13e0fb67d7c15d92b26a181cf5983
PageCount 12
ParticipantIDs ieee_primary_5431780
PublicationCentury 2000
PublicationDate 2009-11
PublicationDateYYYYMMDD 2009-11-01
PublicationDate_xml – month: 11
  year: 2009
  text: 2009-11
PublicationDecade 2000
PublicationTitle 2009 IEEE/ACM International Conference on Automated Software Engineering
PublicationTitleAbbrev ASE
PublicationYear 2009
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0000452981
ssj0051577
ssj0001130041
Score 1.9652804
Snippet We investigate claims and assumptions made in several recent papers about fault localization (FL) techniques. Most of these claims have to do with evaluating...
SourceID ieee
SourceType Publisher
StartPage 76
SubjectTerms Computer science
Costs
Data analysis
Data mining
Fault localization
Genetic mutations
Humans
mutation analysis
Programming profession
Software engineering
Software testing
Title Evaluating the Accuracy of Fault Localization Techniques
URI https://ieeexplore.ieee.org/document/5431780
WOSCitedRecordID wos000278137400007&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwGA1zePA0dRN_k4NH69o0bdLjkA0PYwycsNtIv3wRQTrZWsH_3iRLpwcv0kt_XErKl_fype89Qu5KgMLYIxLAsojnJo-ktJNhirb0II1lqbxQeCpmM7lcFvMOud9rYRDR_3yGD-7U7-XrNTSuVTZ0um0h7QL9QAix02rt-ynOGrwIGkrfX0m8l1Q7K1vY9imMlq_IiNtHrcgrs_xftN5P4ToOQr4kLoaj5_HO1dIFwf8KYPH4M-n9782PyeBHyEfne4g6IR2sTkmvTXKgobD7RI6D6Xf1Si0jpCOAZqPgi64NnajmvaZTh3lBs0kXrfHrdkBeJuPF41MUMhUiZSu7jnSZWwjX2q-LSgZGC1vXxvIqwS1XEwUqlaQYmzIXWkCS6YKVLFeWCIDJCpmekW61rvCcUB6j0Bw4cjCcm0yBQoY5i5lhkCJekL4biNXHzjZjFcbg8u_bV-TIb9R4md816dabBm_IIXzWb9vNrf_W3_KyoRE
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5jCnqauom_zcGjdW2aNulxyMbEOgZO2G2kry8iSCdbK_jfm2bp9OBFeknbS0h5-d576fd9hNxkAIk2lyeARR6PdexJaTbDEE3oQejLTFmicComEzmfJ9MWud1yYRDR_nyGd_XQnuXnS6jqVlm_5m0LaQr0nYhzFmzYWtuOSi0OnjgWpe2wBFZNqtmXDXBbH0aTsUiPm1cNzSsyFYBo1J_cve-ofIGf9AfPw42uZW0F_8uCxSLQqPO_uR-Q3g-Vj063IHVIWlgckU7j5UBdaHeJHDrZ7-KVmpyQDgCqlYIvutR0pKr3kqY16jnWJp010q_rHnkZDWf3Y8-5KnjKxHbp5VlsQDzPbWWUMdC5MJGtTWYluMnWRIJKBSH6OotFLiCI8oRlLFYmFQAdJTI8Ju1iWeAJodxHkXPgyEFzriMFChnGzGeaQYh4Srr1Qiw-NsIZC7cGZ38_viZ749lTukgfJo_nZN8e21jS3wVpl6sKL8kufJZv69WV_e7fjS2kWA
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2009+IEEE%2FACM+International+Conference+on+Automated+Software+Engineering&rft.atitle=Evaluating+the+Accuracy+of+Fault+Localization+Techniques&rft.au=Ali%2C+S.&rft.au=Andrews%2C+J.H.&rft.au=Dhandapani%2C+T.&rft.au=Wantao+Wang&rft.date=2009-11-01&rft.pub=IEEE&rft.isbn=9781424452590&rft.issn=1938-4300&rft.spage=76&rft.epage=87&rft_id=info:doi/10.1109%2FASE.2009.89&rft.externalDocID=5431780
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1938-4300&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1938-4300&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1938-4300&client=summon