Manifold for Machine Learning Assurance

The increasing use of machine-learning (ML) enabled systems in critical tasks fuels the quest for novel verification and validation techniques yet grounded in accepted system assurance principles. In traditional system development, model-based techniques have been widely adopted, where the central p...

Full description

Saved in:

Bibliographic Details
Published in:	Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results pp. 97 - 100
Main Authors:	Byun, Taejoon, Rayadurgam, Sanjai
Format:	Conference Proceeding
Language:	English
Published:	ACM 01.10.2020
Subjects:	Machine learning machine learning testing Manifolds Monitoring neural networks Software testing Task analysis Test pattern generators Training data variational autoencoder
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	The increasing use of machine-learning (ML) enabled systems in critical tasks fuels the quest for novel verification and validation techniques yet grounded in accepted system assurance principles. In traditional system development, model-based techniques have been widely adopted, where the central premise is that abstract models of the required system provide a sound basis for judging its implementation. We posit an analogous approach for ML systems using an ML technique that extracts from the high-dimensional training data implicitly describing the required system, a low-dimensional underlying structure-a manifold. It is then harnessed for a range of quality assurance tasks such as test adequacy measurement, test input generation, and runtime monitoring of the target ML system. The approach is built on variational autoencoder, an unsupervised method for learning a pair of mutually near-inverse functions between a given high-dimensional dataset and a low-dimensional representation. Preliminary experiments establish that the proposed manifold-based approach, for test adequacy drives diversity in test data, for test generation yields fault-revealing yet realistic test cases, and for run-time monitoring provides an independent means to assess trustability of the target system's output. Ccs Concepts * Software and its engineering → Software testing and debugging; * Computing methodologies → Machine learning.
DOI:	10.1145/3377816.3381734