Abstraction-Aware Inference of Metamorphic Relations

Metamorphic testing is a valuable technique that helps in dealing with the oracle problem. It involves testing software against specifications of its intended behavior given in terms of so called metamorphic relations, statements that express properties relating different software elements (e.g., di...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of the ACM on software engineering Vol. 1; no. FSE; pp. 450 - 472
Main Authors: Nolasco, Agustín, Molina, Facundo, Degiovanni, Renzo, Gorla, Alessandra, Garbervetsky, Diego, Papadakis, Mike, Uchitel, Sebastian, Aguirre, Nazareno, Frias, Marcelo F.
Format: Journal Article
Language:English
Published: New York, NY, USA ACM 12.07.2024
Subjects:
Dynamic analysis
Program specifications
Software and its engineering
Software testing and debugging
Theory of computation
grammar-based fuzzing
oracle problem
Metamorphic testing
ISSN:2994-970X, 2994-970X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Metamorphic testing is a valuable technique that helps in dealing with the oracle problem. It involves testing software against specifications of its intended behavior given in terms of so called metamorphic relations, statements that express properties relating different software elements (e.g., different inputs, methods, etc). The effective application of metamorphic testing strongly depends on identifying suitable domain-specific metamorphic relations, a challenging task, that is typically manually performed. This paper introduces MemoRIA, a novel approach that aims at automatically identifying metamorphic relations. The technique focuses on a particular kind of metamorphic relation, which asserts equivalences between methods and method sequences. MemoRIA works by first generating an object-protocol abstraction of the software being tested, then using fuzzing to produce candidate relations from the abstraction, and finally validating the candidate relations through run-time analysis. A SAT-based analysis is used to eliminate redundant relations, resulting in a concise set of metamorphic relations for the software under test. We evaluate our technique on a benchmark consisting of 22 Java subjects taken from the literature, and compare MemoRIA with the metamorphic relation inference technique SBES. Our results show that by incorporating the object protocol abstraction information, MemoRIA is able to more effectively infer meaningful metamorphic relations, that are also more precise, compared to SBES, measured in terms of mutation analysis. Also, the SAT-based reduction allows us to significantly reduce the number of reported metamorphic relations, while in general having a small impact in the bug finding ability of the corresponding obtained relations.
ISSN:2994-970X
2994-970X
DOI:10.1145/3643747