How Many of All Bugs Do We Find? A Study of Static Bug Detectors

Static bug detectors are becoming increasingly popular and are widely used by professional software developers. While most work on bug detectors focuses on whether they find bugs at all, and on how many false positives they report in addition to legitimate warnings, the inverse question is often neg...

Full description

Saved in:
Bibliographic Details
Published in:2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) pp. 317 - 328
Main Authors: Habib, Andrew, Pradel, Michael
Format: Conference Proceeding
Language:English
Published: ACM 03.09.2018
Subjects:
bug finding
Computer bugs
Defects4J
Detectors
Java
Manuals
Pattern matching
Software
Software engineering
Static analysis
static bug checkers
Testing
ISSN:2643-1572
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Static bug detectors are becoming increasingly popular and are widely used by professional software developers. While most work on bug detectors focuses on whether they find bugs at all, and on how many false positives they report in addition to legitimate warnings, the inverse question is often neglected: How many of all real-world bugs do static bug detectors find? This paper addresses this question by studying the results of applying three widely used static bug detectors to an extended version of the Defects4J dataset that consists of 15 Java projects with 594 known bugs. To decide which of these bugs the tools detect, we use a novel methodology that combines an automatic analysis of warnings and bugs with a manual validation of each candidate of a detected bug. The results of the study show that: (i) static bug detectors find a non-negligible amount of all bugs, (ii) different tools are mostly complementary to each other, and (iii) current bug detectors miss the large majority of the studied bugs. A detailed analysis of bugs missed by the static detectors shows that some bugs could have been found by variants of the existing detectors, while others are domain-specific problems that do not match any existing bug pattern. These findings help potential users of such tools to assess their utility, motivate and outline directions for future work on static bug detection, and provide a basis for future comparisons of static bug detection with other bug finding techniques, such as manual and automated testing.
AbstractList Static bug detectors are becoming increasingly popular and are widely used by professional software developers. While most work on bug detectors focuses on whether they find bugs at all, and on how many false positives they report in addition to legitimate warnings, the inverse question is often neglected: How many of all real-world bugs do static bug detectors find? This paper addresses this question by studying the results of applying three widely used static bug detectors to an extended version of the Defects4J dataset that consists of 15 Java projects with 594 known bugs. To decide which of these bugs the tools detect, we use a novel methodology that combines an automatic analysis of warnings and bugs with a manual validation of each candidate of a detected bug. The results of the study show that: (i) static bug detectors find a non-negligible amount of all bugs, (ii) different tools are mostly complementary to each other, and (iii) current bug detectors miss the large majority of the studied bugs. A detailed analysis of bugs missed by the static detectors shows that some bugs could have been found by variants of the existing detectors, while others are domain-specific problems that do not match any existing bug pattern. These findings help potential users of such tools to assess their utility, motivate and outline directions for future work on static bug detection, and provide a basis for future comparisons of static bug detection with other bug finding techniques, such as manual and automated testing.
Author Habib, Andrew
Pradel, Michael
Author_xml – sequence: 1
  givenname: Andrew
  surname: Habib
  fullname: Habib, Andrew
  organization: TU Darmstadt,Department of Computer Science,Germany
– sequence: 2
  givenname: Michael
  surname: Pradel
  fullname: Pradel, Michael
  organization: TU Darmstadt,Department of Computer Science,Germany
BookMark eNotj01PwlAQRZ9GEwFZu3Dz_kBx3ndnpRUETDAu0OiODGVqampr2kcM_15QVie59-Qmty_O6qZmIa4UjJSy7sZokyobRgdqZU5Ef5-CcWjC-6noaW9NolzQF2LYdZ8AoNMATrueuJs3P_KJ6p1sCplVlbzffnRy0sg3ltOy3tzKTC7jdvPXLyPFMj8ocsKR89i03aU4L6jqeHjkQLxOH17G82TxPHscZ4uEjFUxQQ7FGpFTYkWIniisiQsGBK1tSgrBOSAkXyBhjj6wpdwrFWzK1ngzENf_uyUzr77b8ova3Qr3XyBY8wtbkkgB
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1145/3238147.3238213
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 145035937X
9781450359375
EISSN 2643-1572
EndPage 328
ExternalDocumentID 9000074
Genre orig-research
GroupedDBID 29I
6IE
6IF
6IH
6IK
6IL
6IM
6IN
6J9
AAJGR
AAWTH
ABLEC
ACREN
ADYOE
ADZIZ
AFYQB
ALMA_UNASSIGNED_HOLDINGS
AMTXH
APO
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
M43
OCL
RIE
RIL
ID FETCH-LOGICAL-a341t-9e7fb99e8ae1a996aa7baefe0902248a190550a9a6f9a9c967e4ac611748e4363
IEDL.DBID RIE
ISICitedReferencesCount 71
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000553784500032&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 13 06:22:43 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a341t-9e7fb99e8ae1a996aa7baefe0902248a190550a9a6f9a9c967e4ac611748e4363
PageCount 12
ParticipantIDs ieee_primary_9000074
PublicationCentury 2000
PublicationDate 2018-09-03
PublicationDateYYYYMMDD 2018-09-03
PublicationDate_xml – month: 09
  year: 2018
  text: 2018-09-03
  day: 03
PublicationDecade 2010
PublicationTitle 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE)
PublicationTitleAbbrev ASE
PublicationYear 2018
Publisher ACM
Publisher_xml – name: ACM
SSID ssj0002870525
ssj0051577
Score 2.435693
Snippet Static bug detectors are becoming increasingly popular and are widely used by professional software developers. While most work on bug detectors focuses on...
SourceID ieee
SourceType Publisher
StartPage 317
SubjectTerms bug finding
Computer bugs
Defects4J
Detectors
Java
Manuals
Pattern matching
Software
Software engineering
Static analysis
static bug checkers
Testing
Title How Many of All Bugs Do We Find? A Study of Static Bug Detectors
URI https://ieeexplore.ieee.org/document/9000074
WOSCitedRecordID wos000553784500032&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09TwJBEN0gsbBCBeN3trB04eD2aypFkdBIKDTSkbljzpAQzsChf9_d5QQLG6vb3F2x2c3kzZt5M8PYDUqpkixWAiI0wvn_SiTYIaFMChlRrON4GoZNmOHQjscwqrDbbS0MEQXxGTX9MuTyp3m69qGyFoTEm9xje8boTa3WNp7iE3Zq12fPwbQxZSuftlSt2EOTNE3_7PhZBr9mqQQo6df-t4lD1tjV5PHRFm2OWIUWx6z2M5SBlzZaZ_eD_Is_OxPneca78zl_WL-veC_nb8T7joDf8S734sHw3buas9T_wntUhAD-qsFe-08vjwNRjkkQ6CCoEEAmSwDIIrXR0RdEkyBl5BWXHWnRQb6jIQioM0BIQRuSmOq24yKWpLuNE1Zd5As69TonmhqtiSJnphYM2gjIcUAgpW0kkzNW9wcy-dh0wpiUZ3H-9-sLduDci6CuiOJLVi2Wa7pi--lnMVstr8P1fQP_7JcI
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT8JAEJ0gmugJFYzf7sGjhUJ3u7snRZFgBMIBIzcyLVNDQqiBon_f3bWCBy-euml72Oxm8ubjzTyAa-RcREkgPO2j9Iz_L7wIG-QJGeuEKAiDYOLEJmS_r0YjPSjAzboXhogc-Yyqdulq-ZM0XtlUWU27whvfgm2rnJV3a60zKrZkJzaT9gxQS5kP86lzUQssOHFZtc-GVTP4pabiwKRd-t829qGy6cpjgzXeHECB5odQ-pFlYLmVluGuk36ynjFyliasOZux-9XbkrVS9kqsbULwW9Zklj7ovltncxrbX1iLMpfCX1bgpf04fOh4uVCChwaEMk-TTCKtSSHV0QQwiDJCSshyLhtcoQF9E4igxjDRqGMdSuIYh3UTjSji5j6OoDhP53RsmU40kWFI5BtDVVqi8jWZKFCTCJXPoxMo2wMZv3_PwhjnZ3H69-sr2O0Me91x96n_fAZ7xtlwXAs_OIditljRBezEH9l0ubh0V_kFFJmaUQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=2018+33rd+IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%28ASE%29&rft.atitle=How+Many+of+All+Bugs+Do+We+Find%3F+A+Study+of+Static+Bug+Detectors&rft.au=Habib%2C+Andrew&rft.au=Pradel%2C+Michael&rft.date=2018-09-03&rft.pub=ACM&rft.eissn=2643-1572&rft.spage=317&rft.epage=328&rft_id=info:doi/10.1145%2F3238147.3238213&rft.externalDocID=9000074