Feature Guided Masked Autoencoder for Self-supervised Learning in Remote Sensing

Self-supervised learning guided by masked image modeling, such as Masked AutoEncoder (MAE), has attracted wide attention for pretraining vision transformers in remote sensing. However, MAE tends to excessively focus on pixel details, limiting the model's capacity for semantic understanding, par...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of selected topics in applied earth observations and remote sensing Vol. 18; pp. 1 - 17
Main Authors: Wang, Yi, Hernandez, Hugo Hernandez, Albrecht, Conrad M, Zhu, Xiao Xiang
Format: Journal Article
Language:English
Published: Piscataway IEEE 01.01.2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Earth
Earth observation
Feature extraction
geospatial foundation models
Histograms
Image edge detection
Image processing
Image reconstruction
Indexes
masked autoencoder
masked autoencoder (MAE)
Noise
Pixels
Radar imaging
Radar polarimetry
Remote sensing
remote sensing (RS)
SAR (radar)
Self-supervised learning
Synthetic aperture radar
ISSN:1939-1404, 2151-1535
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Self-supervised learning guided by masked image modeling, such as Masked AutoEncoder (MAE), has attracted wide attention for pretraining vision transformers in remote sensing. However, MAE tends to excessively focus on pixel details, limiting the model's capacity for semantic understanding, particularly for noisy Synthetic Aperture Radar (SAR) images. In this paper, we explore spectral and spatial remote sensing image features as improved MAE-reconstruction targets. We first conduct a study on reconstructing various image features, all performing comparably well or better than raw pixels. Based on such observations, we propose Feature Guided Masked Autoencoder (FG-MAE): reconstructing a combination of Histograms of Oriented Gradients (HOG) and Normalized Difference Indices (NDI) for multispectral images, and reconstructing HOG for SAR images. Experimental results on three downstream tasks illustrate the effectiveness of FG-MAE with a particular boost for SAR imagery (e.g. up to 5% better than MAE on EuroSAT-SAR). Furthermore, we demonstrate the well-inherited scalability of FG-MAE and release a first series of pretrained vision transformers for medium-resolution SAR and multispectral images.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1939-1404
2151-1535
DOI:10.1109/JSTARS.2024.3493237