Leveraging metamorphic testing to automatically detect inconsistencies in code generator families

SUMMARY Generative software development has paved the way for the creation of multiple code generators that serve as a basis for automatically generating code to different software and hardware platforms. In this context, the software quality becomes highly correlated to the quality of code generato...

Full description

Saved in:
Bibliographic Details
Published in:Software testing, verification & reliability Vol. 30; no. 1
Main Authors: Boussaa, Mohamed, Barais, Olivier, Sunyé, Gerson, Baudry, Benoit
Format: Journal Article
Language:English
Published: Chichester Wiley Subscription Services, Inc 01.01.2020
Wiley
Subjects:
code generators
Computation and Language
Computer Science
Computer software selection and evaluation
Confidence
Generators
metamorphic testing
Non functional properties
Performance measurement
Software design
Software development
Software quality
Software reliability
Software testing
Statistical methods
test automation
test oracle
Test oracles
Testing
non-functional properties
test oracle
software quality
code generators
test automation
metamorphic testing
ISSN:0960-0833, 1099-1689, 1099-1689
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:SUMMARY Generative software development has paved the way for the creation of multiple code generators that serve as a basis for automatically generating code to different software and hardware platforms. In this context, the software quality becomes highly correlated to the quality of code generators used during software development. Eventual failures may result in a loss of confidence for the developers, who will unlikely continue to use these generators. It is then crucial to verify the correct behaviour of code generators in order to preserve software quality and reliability. In this paper, we leverage the metamorphic testing approach to automatically detect inconsistencies in code generators via so‐called “metamorphic relations”. We define the metamorphic relation (i.e., test oracle) as a comparison between the variations of performance and resource usage of test suites running on different versions of generated code. We rely on statistical methods to find the threshold value from which an unexpected variation is detected. We evaluate our approach by testing a family of code generators with respect to resource usage and performance metrics for five different target software platforms. The experimental results show that our approach is able to detect, among 95 executed test suites, 11 performance and 15 memory usage inconsistencies. This paper shows an approach based on metamorphic testing to automatically detect inconsistencies in code generator families. The metamorphic relation is defined as a comparison between the variations of performance and resource usage of test suites running on different versions of generated code. Statistical methods are applied to find the threshold value from which an unexpected variation is detected. The experimental results show that our approach is able to detect several non‐functional inconsistencies.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0960-0833
1099-1689
1099-1689
DOI:10.1002/stvr.1721