Lightweight Multi-Objective and Many-Objective Problem Formulations for Evolutionary Neural Architecture Search with the Training-Free Performance Metric Synaptic Flow

Neural architecture search (NAS) with naive problem formulations and applications of conventional search algorithms often incur prohibitive search costs due to the evaluations of many candidate architectures. For each architecture, its accuracy performance can be properly evaluated after hundreds (o...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Informatica (Ljubljana) Jg. 47; H. 3; S. 303 - 314
Hauptverfasser: Vo, An, Pham, Tan Ngoc, Nguyen, Van Bich, Luong, Ngoc Hoang
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Ljubljana Slovenian Society Informatika / Slovensko drustvo Informatika 01.09.2023
Schlagworte:
Accuracy
Artificial neural networks
Business metrics
Evolutionary algorithms
Floating point arithmetic
Genetic algorithms
Multiple objective analysis
Network latency
Neural networks
Objectives
Optimization
Performance evaluation
Search algorithms
Sorting algorithms
Training
ISSN:0350-5596, 1854-3871
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Neural architecture search (NAS) with naive problem formulations and applications of conventional search algorithms often incur prohibitive search costs due to the evaluations of many candidate architectures. For each architecture, its accuracy performance can be properly evaluated after hundreds (or thousands) of computationally expensive training epochs are performed to achieve proper network weights. A so-called zero-cost metric, Synaptic Flow, computed based on random network weight values at initialization, is found to exhibit certain correlations with the neural network test accuracy and can thus be used as an efficient proxy performance metric during the search. Besides, NAS in practice often involves not only optimizing for network accuracy performance but also optimizing for network complexity, such as model size, number of floating point operations, or latency, as well. In this article, we study various NAS problem formulations in which multiple aspects of deep neural networks are treated as multiple optimization objectives. We employ a widely-used multi-objective evolutionary algorithm, i.e., the non-dominated sorting genetic algorithm II (NSGA-II), to approximate the optimal Pareto-optimal fronts for these NAS problem formulations. Experimental results on the NAS benchmark NATS-Bench show the advantages and disadvantages of each formulation.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0350-5596
1854-3871
DOI:10.31449/inf.v47i3.4736