Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction

In the research of mining software repositories, we need to label a large amount of data to construct a predictive model. The correctness of the labels will affect the performance of a model substantially. However, limited studies have been performed to investigate the impact of mislabeled instances...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on software engineering Ročník 48; číslo 7; s. 2541 - 2556
Hlavní autoři: Wu, Xiaoxue, Zheng, Wei, Xia, Xin, Lo, David
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 01.07.2022
IEEE Computer Society
Témata:
Case studies
Chromium
Classification
Computer bugs
Cybersecurity
Data models
data quality
Datasets
Debugging
label correctness
Noise measurement
Prediction models
Predictive models
Security
Security bug report prediction
Tuning
ISSN:0098-5589, 1939-3520
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In the research of mining software repositories, we need to label a large amount of data to construct a predictive model. The correctness of the labels will affect the performance of a model substantially. However, limited studies have been performed to investigate the impact of mislabeled instances on a predictive model. To bridge the gap, in this article, we perform a case study on the security bug report (SBR) prediction. We found five publicly available datasets for SBR prediction contains many mislabeled instances, which lead to the poor performance of SBR prediction models of recent studies (e.g., the work of Peters et al. and Shu et al. ). Furthermore, it might mislead the research direction of SBR prediction. In this article, we first improve the label correctness of these five datasets by manually analyzing each bug report, and we find 749 SBRs, which are originally mislabeled as Non-SBRs (NSBRs). We then evaluate the impacts of datasets label correctness by comparing the performance of the classification models on both the noisy (i.e., before our correction) and the clean (i.e., after our correction) datasets. The results show that the cleaned datasets result in improvement in the performance of classification models. The performance of the approaches proposed by Peters et al. and Shu et al. on the clean datasets is much better than on the noisy datasets. Furthermore, with the clean datasets, the simple text classification models could significantly outperform the security keywords-matrix-based approaches applied by Peters et al. and Shu et al.
Bibliografie:ObjectType-Case Study-2
SourceType-Scholarly Journals-1
content type line 14
ObjectType-Feature-4
ObjectType-Report-1
ObjectType-Article-3
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2021.3063727