iNL: Implicit non-local network

The attention mechanism of computer vision represented by a non-local network improves the performance of numerous vision tasks while bringing computational burden for deployment Wang et al. (2018). In this work, we explore to release the inference computation for non-local network by decoupling the...

Full description

Saved in:
Bibliographic Details
Published in:Neurocomputing (Amsterdam) Vol. 482; pp. 50 - 59
Main Authors: Han, Yifeng, Chen, Xi, Zhang, Songjie, Qi, Donglian
Format: Journal Article
Language:English
Published: Elsevier B.V 14.04.2022
Subjects:
Attention
Computation cost
Generalized form
Implicit method
Computation cost
Implicit method
Generalized form
Attention
ISSN:0925-2312, 1872-8286
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The attention mechanism of computer vision represented by a non-local network improves the performance of numerous vision tasks while bringing computational burden for deployment Wang et al. (2018). In this work, we explore to release the inference computation for non-local network by decoupling the training/inference procedure. Specifically, we propose the implicit non-local network (iNL). During training, iNL models the dependency between features across long-range affinities like original non-local blocks; during inference, iNL could be reformulated as only two convolution layers but can rival non-local network. In this way, the computation complexity and the memory costs are reduced. In addition, we take a further step and extend our iNL into a more generalized form, which covers the attentions of different orders in computer vision tasks. iNL brings steady improvements on multiple benchmarks of different vision tasks including classification, detection, and instance segmentation. In the meantime, it provides a brand–new perspective to understand the attention mechanism in deep neural networks.
ISSN:0925-2312
1872-8286
DOI:10.1016/j.neucom.2022.01.047