An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models

Software analytics have empowered software organisations to support a wide range of improved decision-making and policy-making. However, such predictions made by software analytics to date have not been explained and justified. Specifically, current defect prediction models still fail to explain why...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on software engineering Jg. 48; H. 1; S. 166 - 185
Hauptverfasser: Jiarpakdee, Jirayus, Tantithamthavorn, Chakkrit Kla, Dam, Hoa Khanh, Grundy, John
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York IEEE 01.01.2022
IEEE Computer Society
Schlagworte:
Algorithms
Analytical models
Decision analysis
Decision making
defect prediction models
Electric breakdown
Empirical analysis
Explainable software analytics
Lime
model-agnostic techniques
Open source software
Optimization
Prediction algorithms
Prediction models
Predictive models
Quality control
Software
Software algorithms
Software engineering
software quality assurance
ISSN:0098-5589, 1939-3520
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Software analytics have empowered software organisations to support a wide range of improved decision-making and policy-making. However, such predictions made by software analytics to date have not been explained and justified. Specifically, current defect prediction models still fail to explain why models make such a prediction and fail to uphold the privacy laws in terms of the requirement to explain any decision made by an algorithm. In this paper, we empirically evaluate three model-agnostic techniques, i.e., two state-of-the-art Local Interpretability Model-agnostic Explanations technique (LIME) and BreakDown techniques, and our improvement of LIME with Hyper Parameter Optimisation (LIME-HPO). Through a case study of 32 highly-curated defect datasets that span across 9 open-source software systems, we conclude that (1) model-agnostic techniques are needed to explain individual predictions of defect models; (2) instance explanations generated by model-agnostic techniques are mostly overlapping (but not exactly the same) with the global explanation of defect models and reliable when they are re-generated; (3) model-agnostic techniques take less than a minute to generate instance explanations; and (4) more than half of the practitioners perceive that the contrastive explanations are necessary and useful to understand the predictions of defect models. Since the implementation of the studied model-agnostic techniques is available in both Python and R, we recommend model-agnostic techniques be used in the future.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2020.2982385