Lightweight Multi-Objective and Many-Objective Problem Formulations for Evolutionary Neural Architecture Search with the Training-Free Performance Metric Synaptic Flow

Neural architecture search (NAS) with naive problem formulations and applications of conventional search algorithms often incur prohibitive search costs due to the evaluations of many candidate architectures. For each architecture, its accuracy performance can be properly evaluated after hundreds (o...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Informatica (Ljubljana) Ročník 47; číslo 3; s. 303 - 314
Hlavní autoři: Vo, An, Pham, Tan Ngoc, Nguyen, Van Bich, Luong, Ngoc Hoang
Médium: Journal Article
Jazyk:angličtina
Vydáno: Ljubljana Slovenian Society Informatika / Slovensko drustvo Informatika 01.09.2023
Témata:
Accuracy
Artificial neural networks
Business metrics
Evolutionary algorithms
Floating point arithmetic
Genetic algorithms
Multiple objective analysis
Network latency
Neural networks
Objectives
Optimization
Performance evaluation
Search algorithms
Sorting algorithms
Training
ISSN:0350-5596, 1854-3871
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Neural architecture search (NAS) with naive problem formulations and applications of conventional search algorithms often incur prohibitive search costs due to the evaluations of many candidate architectures. For each architecture, its accuracy performance can be properly evaluated after hundreds (or thousands) of computationally expensive training epochs are performed to achieve proper network weights. A so-called zero-cost metric, Synaptic Flow, computed based on random network weight values at initialization, is found to exhibit certain correlations with the neural network test accuracy and can thus be used as an efficient proxy performance metric during the search. Besides, NAS in practice often involves not only optimizing for network accuracy performance but also optimizing for network complexity, such as model size, number of floating point operations, or latency, as well. In this article, we study various NAS problem formulations in which multiple aspects of deep neural networks are treated as multiple optimization objectives. We employ a widely-used multi-objective evolutionary algorithm, i.e., the non-dominated sorting genetic algorithm II (NSGA-II), to approximate the optimal Pareto-optimal fronts for these NAS problem formulations. Experimental results on the NAS benchmark NATS-Bench show the advantages and disadvantages of each formulation.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0350-5596
1854-3871
DOI:10.31449/inf.v47i3.4736