An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models

Software analytics have empowered software organisations to support a wide range of improved decision-making and policy-making. However, such predictions made by software analytics to date have not been explained and justified. Specifically, current defect prediction models still fail to explain why...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on software engineering Ročník 48; číslo 1; s. 166 - 185
Hlavní autori: Jiarpakdee, Jirayus, Tantithamthavorn, Chakkrit Kla, Dam, Hoa Khanh, Grundy, John
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York IEEE 01.01.2022
IEEE Computer Society
Predmet:
Algorithms
Analytical models
Decision analysis
Decision making
defect prediction models
Electric breakdown
Empirical analysis
Explainable software analytics
Lime
model-agnostic techniques
Open source software
Optimization
Prediction algorithms
Prediction models
Predictive models
Quality control
Software
Software algorithms
Software engineering
software quality assurance
ISSN:0098-5589, 1939-3520
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Software analytics have empowered software organisations to support a wide range of improved decision-making and policy-making. However, such predictions made by software analytics to date have not been explained and justified. Specifically, current defect prediction models still fail to explain why models make such a prediction and fail to uphold the privacy laws in terms of the requirement to explain any decision made by an algorithm. In this paper, we empirically evaluate three model-agnostic techniques, i.e., two state-of-the-art Local Interpretability Model-agnostic Explanations technique (LIME) and BreakDown techniques, and our improvement of LIME with Hyper Parameter Optimisation (LIME-HPO). Through a case study of 32 highly-curated defect datasets that span across 9 open-source software systems, we conclude that (1) model-agnostic techniques are needed to explain individual predictions of defect models; (2) instance explanations generated by model-agnostic techniques are mostly overlapping (but not exactly the same) with the global explanation of defect models and reliable when they are re-generated; (3) model-agnostic techniques take less than a minute to generate instance explanations; and (4) more than half of the practitioners perceive that the contrastive explanations are necessary and useful to understand the predictions of defect models. Since the implementation of the studied model-agnostic techniques is available in both Python and R, we recommend model-agnostic techniques be used in the future.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2020.2982385